Git Product home page Git Product logo

local-llm-comparison-colab-ui's Introduction

Local LLM Comparison & Colab Links (WIP)

(Update Nov. 27, 2023) The original goal of the repo was to compare some smaller models (7B and 13B) that can be run on consumer hardware so every model had a score for a set of questions from GPT-4. But I realized that as there are many more capable models appearing, the evaluation and comparison process may not suffice.

Therefore, I'm only putting Colab WebUI links for the newer models and you can try them out yourselves with a few clicks - after all, the effectiveness of a language model relies heavily on its suitability for your specific use case. By trying out the models firsthand, you can assess their performance and determine which one best fits your needs.

Newer Models without Scores:

These models can be run on consumer hardware and are generally good (from Reddit suggestions and my own experience). Try them out yourselves (tap the 'Open in Colab' button)!

Model Link Colab Link Date Added Note
zephyr-7B-beta-GGUF https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF Open In Colab 2023/11/27 Roleplay okay, not censored
OpenHermes-2.5-Mistral-7B-GGUF https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-GGUF Open In Colab 2023/11/27 Roleplay good, not censored
dolphin-2.2.1-mistral-7B-GGUF https://huggingface.co/TheBloke/dolphin-2.2.1-mistral-7B-GGUF Open In Colab 2023/11/27 Roleplay okay, not censored
neural-chat-7B-v3-1-GGUF https://huggingface.co/TheBloke/neural-chat-7B-v3-1-GGUF Open In Colab 2023/11/27 Roleplay okay, not censored; some logic flaws
openchat_3.5-16k-GGUF https://huggingface.co/TheBloke/openchat_3.5-16k-GGUF Open In Colab 2023/11/27 Censored
Starling-LM-7B-alpha-GGUF https://huggingface.co/TheBloke/Starling-LM-7B-alpha-GGUF Open In Colab 2023/11/29 Censored; highly rated on Reddit
Orca-2-7B-GGUF https://huggingface.co/TheBloke/Orca-2-7B-GGUF Open In Colab 2023/11/29 Censored
Orca-2-13B-GGUF https://huggingface.co/TheBloke/Orca-2-13B-GGUF Open In Colab 2023/11/29 Censored, some weird logic flaws worse than 7B version
MythoMist-7B-GGUF https://huggingface.co/TheBloke/MythoMist-7B-GGUF Open In Colab 2023/11/29 Roleplay okay, not censored; some logic flaws
NeuralHermes-2.5-Mistral-7B-GGUF https://huggingface.co/TheBloke/NeuralHermes-2.5-Mistral-7B-GGUF Open In Colab 2023/12/05 Roleplay good, not censored
stablelm-zephyr-3b-GGUF https://huggingface.co/TheBloke/stablelm-zephyr-3b-GGUF Open In Colab 2023/12/11 3B; roleplay ok; not censored; some logic flaws
deepseek-llm-7B-chat-GGUF https://huggingface.co/TheBloke/deepseek-llm-7B-chat-GGUF Open In Colab 2023/12/11 Censored
Mistral-7B-Instruct-v0.2-GGUF https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-GGUF Open In Colab 2023/12/13 Partially censored; role play ok; highly rated on Reddit
Mixtral-8x7B-Instruct-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF Open In Colab 2023/12/13 MOE model; partially censored; role play ok
deepsex-34b-GGUF https://huggingface.co/TheBloke/deepsex-34b-GGUF Open In Colab 2023/12/14 34B; NSFW model
phi-2 https://huggingface.co/microsoft/phi-2 Open In Colab 2023/12/15 2.7B; Base model;
Xwin-MLewd-13B-v0.2-GGUF https://huggingface.co/TheBloke/Xwin-MLewd-13B-v0.2-GGUF Open In Colab 2023/12/15 13B; NSFW model
MythoMax-L2-13B-GGUF https://huggingface.co/TheBloke/MythoMax-L2-13B-GGUF Open In Colab 2023/12/15 13B; Censored; role play ok
LLaMA2-13B-Tiefighter-GGUF https://huggingface.co/TheBloke/LLaMA2-13B-Tiefighter-GGUF Open In Colab 2023/12/15 13B; role play good
LLaMA2-13B-Psyfighter2-GGUF https://huggingface.co/TheBloke/LLaMA2-13B-Psyfighter2-GGUF Open In Colab 2023/12/15 13B; Partially censored; role play ok; recommended on Reddit
Noromaid-13B-v0.1.1-GGUF https://huggingface.co/TheBloke/Noromaid-13B-v0.1.1-GGUF Open In Colab 2023/12/15 13B; NSFW model; roleplay good
dolphin-2.5-mixtral-8x7b-GGUF https://huggingface.co/TheBloke/dolphin-2.5-mixtral-8x7b-GGUF Open In Colab 2023/12/20 MOE model; claims to be uncensored but denies some requests; role play not usable
SOLAR-10.7B-Instruct-v1.0-GGUF https://huggingface.co/TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF Open In Colab 2023/12/21 10.7B; Censored; role play ok
Nous-Hermes-2-SOLAR-10.7B-GGUF https://huggingface.co/TheBloke/Nous-Hermes-2-SOLAR-10.7B-GGUF Open In Colab 2024/01/08 10.7B; Partially Censored; role play good
openchat-3.5-0106-GGUF https://huggingface.co/TheBloke/openchat-3.5-0106-GGUF Open In Colab 2024/01/12 Role play and creative writing good; not censored
Mistral-7B-Instruct-v0.2-code-ft-GGUF https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.2-code-ft-GGUF Open In Colab 2024/01/12 Coding model
Panda-7B-v0.1-GGUF https://huggingface.co/TheBloke/Panda-7B-v0.1-GGUF Open In Colab 2024/01/12 Role play ok; creative writing good; partially censored
LLaMA-Pro-8B-GGUF https://huggingface.co/TheBloke/LLaMA-Pro-8B-GGUF Open In Colab 2024/01/16 Claims to be good at math/coding; quality so so
dolphin-2.6-mistral-7B-dpo-laser-GGUF https://huggingface.co/TheBloke/dolphin-2.6-mistral-7B-dpo-laser-GGUF Open In Colab 2024/01/16 Uncensored; Good quality; role play good
laserxtral-GGUF https://huggingface.co/dagbs/laserxtral-GGUF Open In Colab 2024/01/17 4x7B MOE model; half memory than Mixtral; not censored; role play good
Mixtral_7Bx2_MoE-GGUF https://huggingface.co/TheBloke/Mixtral_7Bx2_MoE-GGUF Open In Colab 2024/01/23 2x7B MOE model; uncensored; role play ok
CapybaraHermes-2.5-Mistral-7B-GGUF https://huggingface.co/TheBloke/CapybaraHermes-2.5-Mistral-7B-GGUF Open In Colab 2024/02/06 Censored;
stable-code-3b-GGUF https://huggingface.co/TheBloke/stable-code-3b-GGUF/ Open In Colab 2024/02/06 3B; Coding model; quality meh probably too small
EstopianMaid-13B-GGUF https://huggingface.co/TheBloke/EstopianMaid-13B-GGUF Open In Colab 2024/02/06 13B; Roleplay model
gemma-7b-it-GGUF https://huggingface.co/mlabonne/gemma-7b-it-GGUF/ Open In Colab 2024/02/28 7B; Censored
StarCoder2-15B-GGUF https://huggingface.co/second-state/StarCoder2-15B-GGUF Open In Colab 2024/03/20 15B; coding model; highly voted on r/LocalLLaMA
Qwen1.5-7B-Chat-GGUF https://huggingface.co/Qwen/Qwen1.5-7B-Chat-GGUF Open In Colab 2024/03/20 7B; Censored; highly voted on r/LocalLLaMA
Qwen1.5-14B-Chat-GGUF https://huggingface.co/Qwen/Qwen1.5-14B-Chat-GGUF Open In Colab 2024/03/20 14B; Censored; highly voted on r/LocalLLaMA
Hermes-2-Pro-Mistral-7B-GGUF https://huggingface.co/NousResearch/Hermes-2-Pro-Mistral-7B-GGUF Open In Colab 2024/03/22 7B; Not censored; role play ok; highly ranked on Huggingface
Nous-Hermes-2-Mistral-7B-DPO-GGUF https://huggingface.co/NousResearch/Nous-Hermes-2-Mistral-7B-DPO-GGUF Open In Colab 2024/03/22 7B; Partially censored; role play good; highly ranked on Huggingface
stable-code-instruct-3b-GGUF https://huggingface.co/bartowski/stable-code-instruct-3b-GGUF Open In Colab 2024/03/27 3B; instruction tuned code generation model
Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 https://huggingface.co/Qwen/Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4 Open In Colab 2024/04/03 MOE; Small footprint; some logical errors
Octopus-v2 https://huggingface.co/NexaAIDev/Octopus-v2/ Open In Colab 2024/04/07 2B; non-quantized; optimized for on-device Android APIs
codegemma-7b-it-GGUF https://huggingface.co/lmstudio-community/codegemma-7b-it-GGUF Open In Colab 2024/04/18 7B; coding model
CodeQwen1.5-7B-Chat-GGUF https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat-GGUF Open In Colab 2024/04/18 7B; coding model
WizardLM-2-7B-GGUF https://huggingface.co/MaziyarPanahi/WizardLM-2-7B-GGUF Open In Colab 2024/04/18 7B; Censored
Meta-Llama-3-8B-Instruct-GGUF https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF Open In Colab 2024/04/19 8B; Censored
dolphin-2.9-llama3-8b-gguff https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-8b-gguff Open In Colab 2024/04/22 8B; Uncensored; logic seems degraded from Llama-3-8B
Lexi-Llama-3-8B-Uncensored-GGUF https://huggingface.co/Orenguteng/Lexi-Llama-3-8B-Uncensored-GGUF Open In Colab 2024/04/24 8B; Uncensored
Llama3-8B-Chinese-Chat-GGUF https://huggingface.co/QuantFactory/Llama3-8B-Chinese-Chat-GGUF Open In Colab 2024/04/24 8B; Chinese
Phi-3-mini-4k-instruct-gguf https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf Open In Colab 2024/04/24 3.8B; Censored; Fast
Llama-3-8B-Instruct-32k-v0.1-GGUF https://huggingface.co/MaziyarPanahi/Llama-3-8B-Instruct-32k-v0.1-GGUF Open In Colab 2024/04/25 8B; 32K context; good for summarization of long text

Older Models with Scores:

These models work better among the models I tested on my hardware (i5-12490F, 32GB RAM, RTX 3060 Ti GDDR6X 8GB VRAM): (Note: Because llama.cpp has made some breaking changes to the support of older ggml models. Some older ggml versions listed below may not work properly on current llama.cpp. But there should be GPTQ equivalents or newer ggml versions for the models.)

NOTE:

  • Major edit on June 30, 2023. Since I noticed GPT-4 started to give more granular scores and tends to give higher scores, I have decided to re-run the tests for all models so they can be compared (All evaluations done with GPT-4-0613 for consistency). The scores in the table below are the average of the scores from the latest tests. The scores in the table below are not comparable to the scores in the previous versions of this README.
Model_Name Avg_Score Colab_Link Date_Added Link
Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) 10.00 Open In Colab 2023/10/08 https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GGUF
Llama-2-13B-chat (using oobabooga/text-generation-webui) 9.65 Open In Colab 2023/07/20 https://huggingface.co/TheBloke/Llama-2-13B-chat-GGML
wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) 9.63 Open In Colab 2023/05/07 https://huggingface.co/TheBloke/wizard-vicuna-13B-GGML
Nous-Capybara-7B (using oobabooga/text-generation-webui) 9.56 Open In Colab 2023/10/08 https://huggingface.co/TheBloke/Nous-Capybara-7B-GGUF
vicuna-13B-v1.5 (using oobabooga/text-generation-webui) 9.53 Open In Colab 2023/08/09 https://huggingface.co/TheBloke/vicuna-13B-v1.5-GGML
wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) 9.53 Open In Colab 2023/05/29 https://huggingface.co/TheBloke/wizardLM-13B-1.0-GPTQ
airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) 9.50 Open In Colab 2023/06/30 https://huggingface.co/TheBloke/airoboros-13B-gpt4-1.4-GPTQ
Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) 9.44 Open In Colab 2023/06/03 https://huggingface.co/TheBloke/Nous-Hermes-13B-GPTQ/tree/main
Dolphin-Llama-13B (using oobabooga/text-generation-webui) 9.38 Open In Colab 2023/07/24 https://huggingface.co/TheBloke/Dolphin-Llama-13B-GGML
Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) 9.37 Open In Colab 2023/10/08 https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF
OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) 9.37 Open In Colab 2023/08/15 https://huggingface.co/TheBloke/OpenOrca-Platypus2-13B-GGML
airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) 9.34 Open In Colab 2023/08/01 https://huggingface.co/TheBloke/airoboros-l2-13b-gpt4-2.0-GGML
Chronos-13B-v2 (using oobabooga/text-generation-webui) 9.31 Open In Colab 2023/08/09 https://huggingface.co/TheBloke/Chronos-13B-v2-GGML
vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) 9.31 Open In Colab 2023/06/29 https://huggingface.co/TheBloke/vicuna-13b-v1.3.0-GPTQ
MythoLogic-13B (using oobabooga/text-generation-webui) 9.31 Open In Colab 2023/07/20 https://huggingface.co/TheBloke/MythoLogic-13B-GGML
Selfee-13B-GPTQ (using oobabooga/text-generation-webui) 9.28 Open In Colab 2023/06/07 https://huggingface.co/TheBloke/Selfee-13B-GPTQ
WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) 9.28 Open In Colab 2023/07/26 https://huggingface.co/TheBloke/WizardLM-13B-V1.2-GGML
minotaur-13B-GPTQ (using oobabooga/text-generation-webui) 9.28 Open In Colab 2023/06/09 https://huggingface.co/TheBloke/minotaur-13B-GPTQ
Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) 9.20 Open In Colab 2023/09/21 https://huggingface.co/TheBloke/Pygmalion-2-13B-SuperCOT2-GGUF
Athena-v1 (using oobabooga/text-generation-webui) 9.19 Open In Colab 2023/08/31 https://huggingface.co/TheBloke/Athena-v1-GGUF
PuddleJumper-13B (using oobabooga/text-generation-webui) 9.1875 Open In Colab 2023/08/29 https://huggingface.co/TheBloke/PuddleJumper-13B-GGUF
Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) 9.17 Open In Colab 2023/07/24 https://huggingface.co/TheBloke/Nous-Hermes-Llama2-GGML
Luban-13B (using oobabooga/text-generation-webui) 9.15 Open In Colab 2023/08/31 https://huggingface.co/TheBloke/Luban-13B-GGUF
MythoBoros-13B (using oobabooga/text-generation-webui) 9.15 Open In Colab 2023/07/24 https://huggingface.co/TheBloke/MythoBoros-13B-GGML
13B-Ouroboros (using oobabooga/text-generation-webui) 9.11 Open In Colab 2023/07/27 https://huggingface.co/TheBloke/13B-Ouroboros-GGML
tulu-13B-GPTQ (using oobabooga/text-generation-webui) 9.06 Open In Colab 2023/06/13 https://huggingface.co/TheBloke/tulu-13B-GPTQ
AlpacaCielo-13B (using oobabooga/text-generation-webui) 9.03125 Open In Colab 2023/07/27 https://huggingface.co/TheBloke/AlpacaCielo-13B-GGML
StableBeluga-13B (using oobabooga/text-generation-webui) 9 Open In Colab 2023/08/01 https://huggingface.co/TheBloke/StableBeluga-13B-GGML
Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) 8.97 Open In Colab 2023/08/10 https://huggingface.co/TheBloke/Chronos-Hermes-13B-v2-GGML
OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) 8.97 Open In Colab 2023/09/05 https://huggingface.co/TheBloke/OpenBuddy-Llama2-13B-v11.1-GGUF
Camel-Platypus2-13B (using oobabooga/text-generation-webui) 8.94 Open In Colab 2023/08/15 https://huggingface.co/TheBloke/Camel-Platypus2-13B-GGML
airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) 8.94 Open In Colab 2023/09/21 https://huggingface.co/TheBloke/airoboros-l2-13b-gpt4-m2.0-GGUF
UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) 8.89 Open In Colab 2023/06/30 https://huggingface.co/TheBloke/UltraLM-13B-GPTQ
13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) 8.88 Open In Colab 2023/06/03 https://huggingface.co/digitous/13B-HyperMantis_GPTQ_4bit-128g/
Stable-Platypus2-13B (using oobabooga/text-generation-webui) 8.875 Open In Colab 2023/08/15 https://huggingface.co/TheBloke/Stable-Platypus2-13B-GGML
Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) 8.84 Open In Colab 2023/05/25 https://huggingface.co/TheBloke/airoboros-13B-GPTQ
Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) 8.84 Open In Colab 2023/09/21 https://huggingface.co/TheBloke/Kuchiki-1.1-L2-7B-GGUF
WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) 8.80625 Open In Colab 2023/08/09 https://huggingface.co/TheBloke/WizardLM-1.0-Uncensored-Llama2-13B-GGML
Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) 8.75 Open In Colab 2023/08/10 https://huggingface.co/TheBloke/Chronos-Beluga-v2-13B-GGML
Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) 8.75 Open In Colab 2023/06/09 https://huggingface.co/TheBloke/Vicuna-13B-CoT-GPTQ
wizardLM-7B.q4_2 (in GPT4All) 8.75 No 2023/05/07 https://gpt4all.io/models/ggml-wizardLM-7B.q4_2.bin
OpenChat_v3.2 (using oobabooga/text-generation-webui) 8.71875 Open In Colab 2023/08/01 https://huggingface.co/TheBloke/OpenChat_v3.2-GGML
Huginn-13B (using oobabooga/text-generation-webui) 8.7125 Open In Colab 2023/08/10 https://huggingface.co/TheBloke/Huginn-13B-GGML
WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) 8.66 Open In Colab 2023/07/17 https://huggingface.co/TheBloke/WizardLM-13B-V1.1-GGML
robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) 8.66 Open In Colab 2023/06/19 https://huggingface.co/TheBloke/robin-13B-v2-GPTQ
llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) 8.625 Open In Colab 2023/07/21 https://huggingface.co/TheBloke/llama-2-13B-Guanaco-QLoRA-GGML
mpt-7b-chat (in GPT4All) 8.53 No 2023/05/11 https://gpt4all.io/models/ggml-mpt-7b-chat.bin
chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) 8.48125 Open In Colab 2023/06/16 https://huggingface.co/TheBloke/chronos-hermes-13B-GPTQ
Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) 8.46875 Open In Colab 2023/07/20 https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGML
stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) 8.25 Open In Colab 2023/05/12 https://huggingface.co/TheBloke/stable-vicuna-13B-GPTQ
manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) 8.21875 Open In Colab 2023/05/24 https://huggingface.co/TheBloke/manticore-13b-chat-pyg-GPTQ
CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) 8.09375 Open In Colab 2023/06/10 https://huggingface.co/TheBloke/CAMEL-13B-Combined-Data-GPTQ
WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) 8.09375 Open In Colab 2023/06/02 https://huggingface.co/TheBloke/WizardLM-Uncensored-Falcon-7B-GPTQ
llama-13b-supercot-GGML (using oobabooga/text-generation-webui) 8.01 Open In Colab 2023/07/05 https://huggingface.co/TheBloke/llama-13b-supercot-GGML
Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) 7.96875 Open In Colab 2023/05/24 https://huggingface.co/TheBloke/Project-Baize-v2-13B-GPTQ
koala-13B-4bit-128g.GGML (using llama.cpp) 7.9375 No 2023/05/07 https://huggingface.co/TheBloke/koala-13B-GPTQ-4bit-128g-GGML
wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) 7.90625 Open In Colab 2023/05/19 https://huggingface.co/4bit/WizardLM-13B-Uncensored-4bit-128g
vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) 7.875 Open In Colab 2023/06/29 https://huggingface.co/TheBloke/vicuna-7B-v1.3-GPTQ
Manticore-13B-GPTQ (using oobabooga/text-generation-webui) 7.78125 Open In Colab 2023/05/23 https://huggingface.co/TheBloke/Manticore-13B-GPTQ
vicuna-13b-1.1-q4_2 (in GPT4All) 7.75 No 2023/05/07 https://gpt4all.io/models/ggml-vicuna-13b-1.1-q4_2.bin
falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) 7.625 Open In Colab 2023/06/02 https://huggingface.co/TheBloke/falcon-7b-instruct-GPTQ
guanaco-13B-GPTQ (using oobabooga/text-generation-webui) 7.5625 Open In Colab 2023/05/26 https://huggingface.co/TheBloke/guanaco-13B-GPTQ
Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) 7.31 Open In Colab 2023/08/31 https://huggingface.co/TheBloke/Mythical-Destroyer-V2-L2-13B-GGUF
Kimiko-v2-13B (using oobabooga/text-generation-webui) 7.25 Open In Colab 2023/08/31 https://huggingface.co/TheBloke/Kimiko-v2-13B-GGUF
orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) 7.0875 Open In Colab 2023/06/28 https://huggingface.co/TheBloke/orca_mini_13B-GGML
Platypus2-13B (using oobabooga/text-generation-webui) 7.03125 Open In Colab 2023/08/15 https://huggingface.co/TheBloke/Platypus2-13B-GGML
Redmond-Puffin-13B (using oobabooga/text-generation-webui) 7.03125 Open In Colab 2023/07/20 https://huggingface.co/TheBloke/Redmond-Puffin-13B-GGML
13B-BlueMethod (using oobabooga/text-generation-webui) 7.025 Open In Colab 2023/07/24 https://huggingface.co/TheBloke/13B-BlueMethod-GGML
mpt-7b-instruct 6.6875 No 2023/05/12 https://huggingface.co/TheBloke/MPT-7B-Instruct-GGML
Kimiko-13B (using oobabooga/text-generation-webui) 6.46875 Open In Colab 2023/08/01 https://huggingface.co/TheBloke/Kimiko-13B-GGML
gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) 6.0625 No 2023/05/07 https://huggingface.co/Bradarr/gpt4-x-alpaca-13b-native-ggml-model-q4_0
minotaur-15B-GPTQ (using oobabooga/text-generation-webui) 5.9375 Open In Colab 2023/06/26 https://huggingface.co/TheBloke/minotaur-15B-GPTQ
baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) 5.90625 Open In Colab 2023/07/05 https://huggingface.co/TheBloke/baichuan-vicuna-7B-GGML
gpt4all-j-v1.3-groovy (in GPT4All) 5.6875 No 2023/05/07 https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin

Many thanks to:
❤️ GPT4ALl: https://github.com/nomic-ai/gpt4all-chat
❤️ llama.cpp: https://github.com/ggerganov/llama.cpp
❤️ oobabooga text generation webui: https://github.com/oobabooga/text-generation-webui
❤️ Colab webui inspired by camenduru: https://github.com/camenduru/text-generation-webui-colab/tree/main
❤️ The Bloke for quantization of the models: https://huggingface.co/TheBloke

Coding models tested & average scores:

(All scores are from GPT-4-0613.)

Model_Name Avg_Scores Colab_Link Date_Added Link
CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) 9.8 Open In Colab 2023/08/28 https://huggingface.co/TheBloke/CodeLlama-13B-oasst-sft-v10-GGUF
WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) 9.5 Open In Colab 2023/08/28 https://huggingface.co/TheBloke/WizardCoder-Python-13B-V1.0-GGUF
Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) 8.4 Open In Colab 2023/07/03 https://huggingface.co/TheBloke/Redmond-Hermes-Coder-GPTQ
CodeUp-Alpha-13B-HF (using oobabooga/text-generation-webui) 7.9 Open In Colab 2023/08/15 https://huggingface.co/TheBloke/CodeUp-Alpha-13B-HF-GGML
starchat-beta-GPTQ (using oobabooga/text-generation-webui) 7.6 Open In Colab 2023/07/04 https://huggingface.co/TheBloke/starchat-beta-GPTQ
wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) 7.3 Open In Colab 2023/07/03 https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ
WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) 7.1 Open In Colab 2023/07/21 https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.1-GPTQ
CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) 7 Open In Colab 2023/08/28 https://huggingface.co/TheBloke/CodeLlama-13B-Instruct-GGUF
CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) 6 Open In Colab 2023/08/03 https://huggingface.co/TheBloke/CodeUp-Llama-2-13B-Chat-HF-GGML
WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) 5.9 Open In Colab 2023/07/03 https://huggingface.co/TheBloke/WizardCoder-15B-1.0-GPTQ
wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) 5.9 Open In Colab 2023/05/29 https://huggingface.co/TheBloke/wizardLM-13B-1.0-GPTQ

Questions and scores

Original responses can be found at: https://docs.google.com/spreadsheets/d/1ogDXUiaBx3t7EpMo44aaA6U6kLXX0x2tGRgLg8CISGs/edit?usp=sharing

Question 1: Translate the following English text into French: "The sun rises in the east and sets in the west."

Task Domain: Translation
Expected Good Response: "Le soleil se lève à l'est et se couche à l'ouest."
Explanation: This task tests the model's ability to understand and accurately translate text between languages.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 1
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 8
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 7
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 1
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 1
  • mpt-7b-instruct : 1
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 7
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 8
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 8
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 8
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 1
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 7
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 8
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 1
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 8
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 7
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 8
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 1
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 8.5
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 6
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 7.5
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 4
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 9
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 7.5
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 7.5
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 9.5
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 9.5
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 9.5
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 9.5

Question 2: Summarize the following text: "The water cycle is a natural process that involves the continuous movement of water on, above, and below the Earth's surface. It includes various stages like evaporation, condensation, precipitation, and runoff. This cycle plays a crucial role in maintaining Earth's water balance and supporting life."

Task Domain: Summary
Expected Good Response: "The water cycle is the continuous movement of water on Earth, crucial for maintaining water balance and supporting life."
Explanation: This task evaluates the model's ability to extract the main points from a given text and generate a concise summary.

  • wizardLM-7B.q4_2 (in GPT4All) : 5
  • gpt4all-j-v1.3-groovy (in GPT4All) : 7.5
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 9
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 4
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 7.5
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 1
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 9
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 8.5
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 7.5
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 9
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 9
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 9
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 5
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 9
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 9
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 9
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 8
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 1
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 9
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 9
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 8
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 7.5
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 1
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 1
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 7
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 7.5
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 9
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 7.5
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 7.5
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 9
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 5
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 7.5
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 9
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 9.5
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 9
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 5
  • Athena-v1 (using oobabooga/text-generation-webui) : 9
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 9.5
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 9
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 3: I want you to act as a senior software developer with deep knowledge in system design, frontend programming, and backend programming. Provide a high level design of a mental health journal app. Include the frontend and backend components. Do not write code.

Task Domain: App Design
Explanation: This task evaluates the model's ability to closely follow user's instruction for a complex task.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 9
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 7.5
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 6.5
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 8
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 9
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 10
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 9
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 9.5
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 3
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 9.5
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 8.5
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 7.4
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 4: What are the main causes of the French Revolution according to this passage: "The French Revolution, which took place between 1789 and 1799, was rooted in financial crises, social inequality, and Enlightenment ideas."

Task Domain: Abstractive Question Answering
Expected Good Response: Financial crises, social inequality, and Enlightenment ideas.
Explanation: This task tests the model's ability to understand the context and generate an answer in its own words.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 10
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 1
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 6
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 9.5
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 9
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 9.5
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 7.5
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 8
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 9
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 5: In the following text, identify the two artists mentioned: "During the Renaissance, Leonardo da Vinci and Michelangelo were two of the most influential artists who created masterpieces that continue to inspire people today."

Task Domain: Extractive Question Answering
Expected Good Response: Leonardo da Vinci, Michelangelo
Explanation: This task assesses the model's ability to extract specific information from a given text.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 10
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 9
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 9.5
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 8.5
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 6: Determine the sentiment of this customer review: "I had a fantastic experience at this restaurant. The food was delicious, and the service was outstanding."

Task Domain: Sentiment Analysis
Expected Good Response: Positive
Explanation: This task evaluates the model's ability to analyze text and identify the sentiment expressed.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 10
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 1
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 4
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 8.5
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 5
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 7: Classify the following text into one of these categories: Sports, Technology, Health, or Politics: "Apple recently unveiled its latest iPhone, featuring a faster processor and improved battery life."

Task Domain: Zero-shot Classification
Expected Good Response: Technology
Explanation: This task tests the model's ability to classify text into predefined categories without any prior training on the specific categories.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 5
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 1
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 10
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 2
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 10
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 8: Complete the following sentence by filling in the blank: "The capital city of France is _______."

Task Domain: Mask Filling
Expected Good Response: Paris
Explanation: This task assesses the model's ability to understand context and generate appropriate words to fill in missing information.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 10
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 5
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 10
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 1
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 5
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 9: Write a rhyming couplet about nature.

Task Domain: Poetry Generation
Expected Good Response: "In nature's beauty, we find respite and grace, A symphony of colors that time cannot erase."
Explanation: This task tests the model's ability to generate creative and coherent text that adheres to specific constraints, such as rhyme and theme.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 7.5
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 1
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 6.5
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 6
  • mpt-7b-instruct : 2
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 8
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 9
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 6.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 9
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 4
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 6
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 7
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 7.5
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 7
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 4
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 4
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 8
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 9
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 4
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 7
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 7.5
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 4
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 8
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 7
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 8
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 7.5
  • Kimiko-13B (using oobabooga/text-generation-webui) : 6.5
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 8.5
  • Platypus2-13B (using oobabooga/text-generation-webui) : 6
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 7.5
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 4
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 3
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 7.4
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 8
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 6.5
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 9

Question 10: Based on the following statement, determine if the author's opinion is for or against nuclear energy: "Nuclear energy is a powerful source, but the potential risks and radioactive waste management issues make it a dangerous choice."

Task Domain: Opinion Detection
Expected Good Response: Against
Explanation: This task evaluates the model's ability to understand and identify the author's stance or opinion on a specific topic.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 4
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 9.5
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 6
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 8.5
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 7.5
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 9.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 9.5
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 10
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 4
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 4
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 6.5
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 7
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 8.5
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 8.5
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 3
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 8.5
  • Platypus2-13B (using oobabooga/text-generation-webui) : 9
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 9.5
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 11: Rewrite the following sentence in passive voice: "The dog chased the cat."

Task Domain: Text Rewriting
Expected Good Response: "The cat was chased by the dog."
Explanation: This task tests the model's ability to manipulate and rewrite text according to specific grammatical requirements.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 3
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 10
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 6.5
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 9.5
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 7
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 7.34
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 12: Use the retrieved context to answer a question. Context does not contain the answer. (Prompt too long. See spreadsheet for original prompt)

Task Domain: Document Question Answering
Expected Good Response: "I don't know."
Explanation: This task tests the model's ability to understand context and answer questions based on the information provided in the context.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 1
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 1
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 1
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 1
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 1
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 3
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 1
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 1
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 1
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 1
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 7
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 1
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 1
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 1
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 1
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 1
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 1
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 2
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 1
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 13: Use the retrieved context to answer a question. Context contains the answer. (Prompt too long. See spreadsheet for original prompt)

Task Domain: Document Question Answering
Expected Good Response: "["Semantic text search", "Generative question-answering", "Hybrid search", "Image similarity search", "Product recommendations"]"
Explanation: This task tests the model's ability to understand context and answer questions based on the information provided in the context.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 1
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 1
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 4
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 5.5
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 4
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 1
  • mpt-7b-instruct : 10
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 5
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 2
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 5
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 7
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 8
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 5
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 1
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 8.5
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 5.7
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 6.5
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 7.4
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 5
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 1
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 7.7
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 5.6875
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 1
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 1
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 7.5
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 7.4
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 5.6875
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 7.4
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 7.34
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 1
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 10
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 1
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-13B (using oobabooga/text-generation-webui) : 3
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 7.5
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 1
  • Huginn-13B (using oobabooga/text-generation-webui) : 8.5
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 7.5
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 5
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 1
  • Platypus2-13B (using oobabooga/text-generation-webui) : 8
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 7.4
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 4
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 2
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 2
  • Athena-v1 (using oobabooga/text-generation-webui) : 3
  • Luban-13B (using oobabooga/text-generation-webui) : 1
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 1
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 3
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 14: What is the square root of banana?

Task Domain: Mathematical Reasoning
Expected Good Response: "The question is nonsensical, as square roots can only be calculated for numbers, not for objects or words like 'banana'."
Explanation: This task tests the model's ability to recognize questions that are illogical or nonsensical and respond accordingly.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 1
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 1
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 1
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 1
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 1
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 10
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 10
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 1
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 10
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 1
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 4
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 1
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 2
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 4.5
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 4
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 10
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 1
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 10
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 8.5
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 1
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 7.25
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 8.5
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 4
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 7.5
  • Kimiko-13B (using oobabooga/text-generation-webui) : 1
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 9.5
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 7.5
  • Huginn-13B (using oobabooga/text-generation-webui) : 6.5
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 4.5
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 5.5
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 10
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 1
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 7.5
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 1
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 7.5
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 9.5
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 6.5
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 4

Question 15: Extract the sender's name and address from the following text: "Dear Troy, Thanks for sharing your thoughts on document qa with Claude LLM and your comments on Tim's thoughts. My address is 5000 Forbes Ave, Pittsburgh, PA 15213. Best, Alex." Respond in JSON with one field for name and the other field for address.

Task Domain: Information Extraction
Expected Good Response: {"name": "Alex","address": "5000 Forbes Ave, Pittsburgh, PA 15213"}
Explanation: This task tests the model's ability to extract specific information (sender's name and address) from a given text and present the extracted information in a JSON format.

  • wizardLM-7B.q4_2 (in GPT4All) : 10
  • gpt4all-j-v1.3-groovy (in GPT4All) : 10
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 10
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 10
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 10
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 10
  • mpt-7b-chat (in GPT4All) : 10
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • mpt-7b-instruct : 6
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 10
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 8
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 1
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 10
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 8
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 10
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 10
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 10
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 6
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 10
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 10
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 10
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 10
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 7
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 4
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 10
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 10
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 10
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 10
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 4
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 8.5
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 10
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 10
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 10
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 10
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 10
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 4
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 10
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 4
  • Kimiko-13B (using oobabooga/text-generation-webui) : 10
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 10
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 10
  • Huginn-13B (using oobabooga/text-generation-webui) : 3
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 6
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 10
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 4
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 10
  • Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 10
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 10
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 8.5
  • Athena-v1 (using oobabooga/text-generation-webui) : 10
  • Luban-13B (using oobabooga/text-generation-webui) : 10
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 10
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 10
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 10
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 4
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 10
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 10

Question 16: Given the following list of words. Categorize the words into 5 categories by similarity. Give each category a name. Respond in a python dictionary with key as the category name and value as a list of words in that category. List of words: ['Quagmire', 'Luminous', 'Melancholy', 'Perplexed', 'Jubilant', 'Enigmatic', 'Ambiguous', 'Ravenous', 'Obsolete', 'Tenacious', 'Euphoric', 'Wistful', 'Clandestine', 'Insidious', 'Inquisitive', 'Resilient', 'Surreptitious', 'Serendipity', 'Idiosyncratic', 'Juxtaposition']

Task Domain: Categorization
Expected Good Response:
{ "Emotions": ['Melancholy', 'Jubilant', 'Euphoric', 'Wistful'], "Qualities": ['Luminous', 'Tenacious', 'Resilient'], "Mysterious": ['Quagmire', 'Enigmatic', 'Ambiguous', 'Clandestine', 'Surreptitious'], "Inquisitive": ['Perplexed', 'Inquisitive'], "Uncommon": ['Ravenous', 'Obsolete', 'Insidious', 'Serendipity', 'Idiosyncratic', 'Juxtaposition'] }
Explanation: This task tests the model's ability to categorize a list of words into groups based on their similarity and provide appropriate category names. The response is in a Python dictionary format as specified in the question.

  • wizardLM-7B.q4_2 (in GPT4All) : 1
  • gpt4all-j-v1.3-groovy (in GPT4All) : 1
  • vicuna-13b-1.1-q4_2 (in GPT4All) : 4
  • gpt4-x-alpaca-13b-ggml-q4_0 (using llama.cpp) : 3
  • koala-13B-4bit-128g.GGML (using llama.cpp) : 1
  • wizard-vicuna-13B.ggml.q4_0 (using llama.cpp) : 4
  • mpt-7b-chat (in GPT4All) : 3
  • stable-vicuna-13B-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 4
  • mpt-7b-instruct : 7
  • wizard-lm-uncensored-13b-GPTQ-4bit-128g (using oobabooga/text-generation-webui) : 5
  • Manticore-13B-GPTQ (using oobabooga/text-generation-webui) : 3
  • manticore_13b_chat_pyg_GPTQ (using oobabooga/text-generation-webui) : 1
  • Project-Baize-v2-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • Airoboros-13B-GPTQ-4bit (using oobabooga/text-generation-webui) : 6
  • guanaco-13B-GPTQ (using oobabooga/text-generation-webui) : 5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 7
  • falcon-7b-instruct-GPTQ (using oobabooga/text-generation-webui) : 1
  • WizardLM-Uncensored-Falcon-7B-GPTQ (using oobabooga/text-generation-webui) : 3
  • Nous-Hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 7
  • 13B-HyperMantis_GPTQ (using oobabooga/text-generation-webui) : 3
  • Selfee-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • minotaur-13B-GPTQ (using oobabooga/text-generation-webui) : 2
  • Vicuna-13B-CoT-GPTQ (using oobabooga/text-generation-webui) : 2
  • CAMEL_13B_Combined_Data_GPTQ (using oobabooga/text-generation-webui) : 4
  • tulu-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • chronos-hermes-13B-GPTQ (using oobabooga/text-generation-webui) : 3
  • robin-13B-v2-GPTQ (using oobabooga/text-generation-webui) : 4
  • minotaur-15B-GPTQ (using oobabooga/text-generation-webui) : 1
  • orca-mini-13b.ggmlv3.q5_K_M (using oobabooga/text-generation-webui) : 4
  • vicuna-13b-v1.3.0-GPTQ (using oobabooga/text-generation-webui) : 4
  • vicuna-7B-v1.3-GPTQ (using oobabooga/text-generation-webui) : 4
  • UltraLM-13B-GPTQ (using oobabooga/text-generation-webui) : 5
  • airoboros-13B-gpt4-1.4-GPTQ (using oobabooga/text-generation-webui) : 5
  • llama-13b-supercot-GGML (using oobabooga/text-generation-webui) : 4.5
  • baichuan-vicuna-7B-GGML (using oobabooga/text-generation-webui) : 3
  • WizardLM-13B-V1.1 (using oobabooga/text-generation-webui) : 5
  • Llama-2-13B-chat (using oobabooga/text-generation-webui) : 7.4
  • Luna-AI-Llama2-Uncensored (using oobabooga/text-generation-webui) : 4
  • MythoLogic-13B (using oobabooga/text-generation-webui) : 4
  • Redmond-Puffin-13B (using oobabooga/text-generation-webui) : 4
  • llama-2-13B-Guanaco-QLoRA (using oobabooga/text-generation-webui) : 3
  • Dolphin-Llama-13B (using oobabooga/text-generation-webui) : 4
  • Nous-Hermes-Llama2 (using oobabooga/text-generation-webui) : 4
  • 13B-BlueMethod (using oobabooga/text-generation-webui) : 4
  • MythoBoros-13B (using oobabooga/text-generation-webui) : 4
  • WizardLM-13B-V1.2 (using oobabooga/text-generation-webui) : 7.5
  • 13B-Ouroboros (using oobabooga/text-generation-webui) : 4
  • AlpacaCielo-13B (using oobabooga/text-generation-webui) : 4
  • OpenChat_v3.2 (using oobabooga/text-generation-webui) : 4.5
  • airoboros-l2-13b-gpt4-2.0 (using oobabooga/text-generation-webui) : 4
  • StableBeluga-13B (using oobabooga/text-generation-webui) : 5
  • Kimiko-13B (using oobabooga/text-generation-webui) : 3
  • Chronos-13B-v2 (using oobabooga/text-generation-webui) : 3
  • WizardLM-1.0-Uncensored-Llama2-13B (using oobabooga/text-generation-webui) : 7.4
  • Huginn-13B (using oobabooga/text-generation-webui) : 4
  • Chronos-Beluga-v2-13B (using oobabooga/text-generation-webui) : 2
  • Chronos-Hermes-13B-v2 (using oobabooga/text-generation-webui) : 3
  • vicuna-13B-v1.5 (using oobabooga/text-generation-webui) : 8.5
  • Stable-Platypus2-13B (using oobabooga/text-generation-webui) : 4
  • Platypus2-13B (using oobabooga/text-generation-webui) : 2
  • Camel-Platypus2-13B (using oobabooga/text-generation-webui) : 8.5
  • OpenOrca-Platypus2-13B (using oobabooga/text-generation-webui) : 8.5
  • PuddleJumper-13B (using oobabooga/text-generation-webui) : 8.5
  • Kimiko-v2-13B (using oobabooga/text-generation-webui) : 2
  • Mythical-Destroyer-V2-L2-13B (using oobabooga/text-generation-webui) : 7.5
  • Athena-v1 (using oobabooga/text-generation-webui) : 5
  • Luban-13B (using oobabooga/text-generation-webui) : 8.5
  • OpenBuddy-Llama2-13B-v11.1 (using oobabooga/text-generation-webui) : 8.5
  • Kuchiki-1.1-L2-7B (using oobabooga/text-generation-webui) : 1
  • Pygmalion-2-13B-SuperCOT2 (using oobabooga/text-generation-webui) : 7.4
  • airoboros-l2-13b-gpt4-m2.0 (using oobabooga/text-generation-webui) : 3
  • Mistral-7B-OpenOrca (using oobabooga/text-generation-webui) : 10
  • Nous-Capybara-7B (using oobabooga/text-generation-webui) : 4
  • Mistral-7B-Instruct-v0.1 (using oobabooga/text-generation-webui) : 7.4

Coding questions and answers

I decided to test the coding capability of some coding-specific models and top general purpose models. Original responses can be found at: https://docs.google.com/spreadsheets/d/1ogDXUiaBx3t7EpMo44aaA6U6kLXX0x2tGRgLg8CISGs/edit?usp=sharing

Question 1: Implement a Python function that takes in a list of integers and an integer target, and returns a list of pairs whose sum is equal to the target.

Task Domain: Coding
Expected Good Response:

def find_pairs_with_sum(arr, target):
    if not arr:
        return []
    pairs = []
    seen = set()
    for num in arr:
        complement = target - num
        if complement in seen:
            pairs.append((num, complement))
        seen.add(num)
    return pairs

Explanation of the task: This problem tests the candidate's proficiency in Python and their ability to apply efficient algorithms for a common task in software engineering.

  • wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) : 5.5
  • Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) : 10
  • starchat-beta-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 10
  • WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) : 7.5
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 3
  • CodeUp-Alpha-13B-HF (using oobabooga/text-generation-webui) : 6.5
  • WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) : 10
  • CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) : 10

Question 2: Implement a computer vision model in PyTorch that can classify images of handwritten digits using the MNIST dataset.

Task Domain: Computer Vision
Expected Good Response:

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])
trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=100, shuffle=True, num_workers=2)

model = torch.nn.Sequential(
    torch.nn.Linear(28*28, 128),
    torch.nn.ReLU(),
    torch.nn.Linear(128, 64),
    torch.nn.ReLU(),
    torch.nn.Linear(64, 10)
)

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

for epoch in range(10):
    for i, data in enumerate(trainloader, 0):
        inputs, labels = data
        inputs = inputs.view(inputs.size(0), -1)
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

Explanation of the task: This problem tests the candidate's understanding of computer vision, deep learning, and the PyTorch framework, reflecting their ability to create and train models for real-world applications.

  • wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) : 6.5
  • WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) : 9.5
  • starchat-beta-GPTQ (using oobabooga/text-generation-webui) : 9.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 4
  • WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) : 9.5
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 9.5
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 9.5
  • WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) : 8.5

Question 3: Develop a sentiment analysis model using natural language processing (NLP) techniques in Python with the help of the NLTK library.

Task Domain: Natural Language Processing
Expected Good Response:

import nltk
from nltk.sentiment import SentimentIntensityAnalyzer

nltk.download('vader_lexicon')

def analyze_sentiment(text):
    sia = SentimentIntensityAnalyzer()
    sentiment_score = sia.polarity_scores(text)
    return sentiment_score

Explanation of the task: This problem tests the candidate's proficiency in NLP and their ability to apply it to sentiment analysis, a popular subfield with a wide range of applications in social media analysis, chatbots, and more.

  • wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) : 9.5
  • Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • starchat-beta-GPTQ (using oobabooga/text-generation-webui) : 7.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 4
  • WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) : 5
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 5
  • CodeUp-Alpha-13B-HF (using oobabooga/text-generation-webui) : 8.5
  • WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) : 4

Question 4: Implement a SwiftUI view that displays a list of items and allows the user to filter the list by a search query.

Task Domain: iOS Development Expected Good Response:

import SwiftUI

struct FilteredListView: View {
    @State private var searchText = ""

    let items = ["Apple", "Banana", "Cherry", "Date", "Fig", "Grape"]

    var filteredItems: [String] {
        items.filter { searchText.isEmpty || $0.lowercased().contains(searchText.lowercased()) }
    }

    var body: some View {
        NavigationView {
            VStack {
                TextField("Search", text: $searchText)
                    .padding(.horizontal)
                List(filteredItems, id: \.self) { item in
                    Text(item)
                }
            }
            .navigationBarTitle("Fruits")
        }
    }
}

Explanation of the task: This problem tests the candidate's proficiency in SwiftUI, a modern framework for building user interfaces on Apple platforms, and their ability to create a responsive and dynamic UI.

  • wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) : 4
  • WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) : 3
  • Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) : 5.5
  • starchat-beta-GPTQ (using oobabooga/text-generation-webui) : 5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 3
  • WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) : 4
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 4
  • CodeUp-Alpha-13B-HF (using oobabooga/text-generation-webui) : 6.5
  • WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) : 10
  • CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) : 4

Question 5: Design a RESTful API in Flask (Python) that allows users to create, read, update, and delete (CRUD) notes.

Task Domain: Web Development
Expected Good Response:

from flask import Flask, request, jsonify
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///notes.db'
db = SQLAlchemy(app)
ma = Marshmallow(app)

class Note(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    content = db.Column(db.String(200), nullable=False)

class NoteSchema(ma.SQLAlchemyAutoSchema):
    class Meta:
        model = Note

note_schema = NoteSchema()
notes_schema = NoteSchema(many=True)

@app.route('/notes', methods=['POST'])
def create_note():
    content = request.json['content']
    new_note = Note(content=content)
    db.session.add(new_note)
    db.session.commit()
    return note_schema.dump(new_note)

@app.route('/notes', methods=['GET'])
def get_notes():
    all_notes = Note.query.all()
    return notes_schema.dump(all_notes)

@app.route('/notes/<int:id>', methods=['GET'])
def get_note(id):
    note = Note.query.get_or_404(id)
    return note_schema.dump(note)

@app.route('/notes/<int:id>', methods=['PUT'])
def update_note(id):
    note = Note.query.get_or_404(id)
    content = request.json['content']
    note.content = content
    db.session.commit()
    return note_schema.dump(note)

@app.route('/notes/<int:id>', methods=['DELETE'])
def delete_note(id):
    note = Note.query.get_or_404(id)
    db.session.delete(note)
    db.session.commit()
    return note_schema.dump(note)

if __name__ == '__main__':
    app.run()

Explanation of the task: This problem tests the candidate's ability to design and implement a RESTful API using Flask, a popular web framework in Python, and assesses their understanding of CRUD operations and database management.

  • wizard-vicuna-13B-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • WizardCoder-15B-1.0-GPTQ (using oobabooga/text-generation-webui) : 4
  • Redmond-Hermes-Coder-GPTQ (using oobabooga/text-generation-webui) : 9.5
  • starchat-beta-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • wizardLM-13B-1.0-GPTQ (using oobabooga/text-generation-webui) : 8.5
  • WizardCoder-Guanaco-15B-V1.1 (using oobabooga/text-generation-webui) : 9.5
  • CodeUp-Llama-2-13B-Chat-HF (using oobabooga/text-generation-webui) : 8.5
  • CodeUp-Alpha-13B-HF (using oobabooga/text-generation-webui) : 8.5
  • WizardCoder-Python-13B-V1.0 (using oobabooga/text-generation-webui) : 9.5
  • CodeLlama-13B-oasst-sft-v10 (using oobabooga/text-generation-webui) : 10
  • CodeLlama-13B-Instruct (using oobabooga/text-generation-webui) : 8.5

local-llm-comparison-colab-ui's People

Contributors

klipski avatar ronbalt avatar troyanovsky avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

seshakiran rozek hurricanejin decentralised-ai groundbasesoft wsuff luisriverag wsue gavinchen1314 ramstorage tranhoangnguyen03 ai-jie01 ai-ar4s-dev erima2020 bitsnaps gary109 tnguyenqh jasoncaojingren afilimonov arlvr andycar jcarlosneto glaceage redaalarbi sksundaram-learning krishna999 beachhost lolmeup abhishekrai43 say383 vcappuccio shashwatvt blackwhites ilabutk klipski alienfacepalm kuntal-c emanuelriquelme capzulaz3 proalf ronbalt baileypillon girijeshcse fedaykinofdune bimal-mehta techthiyanes soheilkhodayari narcisoperez bmedi mustafaali96 nissan jakecho lutfimuh lpgpa miles-canton octag0no yasamnik caesarchad silasdao gitrjaa meril99 somesh24 tiamat-tech teddyseptiadi goldendawn369 kp-forks evdcush mainmasterdev hertera1 minyang-chen meta-ks patbhakta vtaranti supun156 syaikhipin igeralt abduibasit 13alvone codenfish cahlen ziyanda-tembe chorseng davidgao7 mkotlarz gumus-helio novagaming1964 kendogg09 cybermazinho f901107 dolife trombone1 heyhimansh tundachef stophobia cheehoon-lee silverdevelopper finyellow majidabdelilah zyh1994 kristofleroux

local-llm-comparison-colab-ui's Issues

make coding scores based on unit tests

Hey,

great initiative to track local llms!

Would you be open to talking about how the scores are created?

  • I created some gpt-4 scores in a project in the past and found them not good enough (they would fluctuate based on input sentences with the same meanings, scores somewhat too arbitrary, different days would give different scores for the same input). At least you should pin the gpt-4 version so you have better control when they roll updates to gpt-4

  • For code one could add unit tests to check the created functions

CUDA driver version is insufficient for CUDA runtime version

This only happen with the WizardLM-1.0-Uncensored-Llama2-13B on Google colab
Thank you for taking a look

/content
Selecting previously unselected package libc-ares2:amd64.
(Reading database ... 121730 files and directories currently installed.)
Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.2_amd64.deb ...
Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...
Selecting previously unselected package libaria2-0:amd64.
Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...
Unpacking libaria2-0:amd64 (1.36.0-1) ...
Selecting previously unselected package aria2.
Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...
Unpacking aria2 (1.36.0-1) ...
Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.2) ...
Setting up libaria2-0:amd64 (1.36.0-1) ...
Setting up aria2 (1.36.0-1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.4) ...
/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link

/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link

Cloning into 'text-generation-webui'...
remote: Enumerating objects: 15569, done.
remote: Counting objects: 100% (15569/15569), done.
remote: Compressing objects: 100% (4561/4561), done.
remote: Total 15569 (delta 10901), reused 15433 (delta 10845), pack-reused 0
Receiving objects: 100% (15569/15569), 25.96 MiB | 24.30 MiB/s, done.
Resolving deltas: 100% (10901/10901), done.
Note: switching to 'ebb4f22028316821c3c14ad91b45e01d309f1ff0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

/content/text-generation-webui
Collecting git+https://github.com/huggingface/peft@96c0277a1b9a381b10ab34dbf84917f9b3b992e6 (from -r requirements.txt (line 21))
  Cloning https://github.com/huggingface/peft (to revision 96c0277a1b9a381b10ab34dbf84917f9b3b992e6) to /tmp/pip-req-build-7fre6n6f
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft /tmp/pip-req-build-7fre6n6f
  Running command git rev-parse -q --verify 'sha^96c0277a1b9a381b10ab34dbf84917f9b3b992e6'
  Running command git fetch -q https://github.com/huggingface/peft 96c0277a1b9a381b10ab34dbf84917f9b3b992e6
  Running command git checkout -q 96c0277a1b9a381b10ab34dbf84917f9b3b992e6
  Resolved https://github.com/huggingface/peft to commit 96c0277a1b9a381b10ab34dbf84917f9b3b992e6
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Ignoring bitsandbytes: markers 'platform_system == "Windows"' don't match your environment
Ignoring auto-gptq: markers 'platform_system == "Windows"' don't match your environment
Collecting auto-gptq==0.3.0+cu117 (from -r requirements.txt (line 25))
  Downloading https://github.com/PanQiWei/AutoGPTQ/releases/download/v0.3.0/auto_gptq-0.3.0+cu117-cp310-cp310-linux_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 8.6 MB/s eta 0:00:00
Ignoring exllama: markers 'platform_system == "Windows"' don't match your environment
Collecting exllama==0.0.9+cu117 (from -r requirements.txt (line 27))
  Downloading https://github.com/jllllll/exllama/releases/download/0.0.9/exllama-0.0.9+cu117-cp310-cp310-linux_x86_64.whl (355 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 355.2/355.2 kB 6.7 MB/s eta 0:00:00
Ignoring llama-cpp-python: markers 'platform_system == "Windows"' don't match your environment
Ignoring llama-cpp-python-cuda: markers 'platform_system == "Windows"' don't match your environment
Collecting llama-cpp-python-cuda==0.1.77+cu117 (from -r requirements.txt (line 33))
  Downloading https://github.com/jllllll/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.1.77+cu117-cp310-cp310-linux_x86_64.whl (2.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 16.9 MB/s eta 0:00:00
Collecting accelerate==0.21.0 (from -r requirements.txt (line 1))
  Downloading accelerate-0.21.0-py3-none-any.whl (244 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 244.2/244.2 kB 4.9 MB/s eta 0:00:00
Collecting colorama (from -r requirements.txt (line 2))
  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting datasets (from -r requirements.txt (line 3))
  Downloading datasets-2.16.1-py3-none-any.whl (507 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 507.1/507.1 kB 10.5 MB/s eta 0:00:00
Collecting einops (from -r requirements.txt (line 4))
  Downloading einops-0.7.0-py3-none-any.whl (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 7.7 MB/s eta 0:00:00
Collecting fastapi==0.95.2 (from -r requirements.txt (line 5))
  Downloading fastapi-0.95.2-py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 6.8 MB/s eta 0:00:00
Collecting gradio_client==0.2.5 (from -r requirements.txt (line 6))
  Downloading gradio_client-0.2.5-py3-none-any.whl (288 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 288.1/288.1 kB 10.5 MB/s eta 0:00:00
Collecting gradio==3.33.1 (from -r requirements.txt (line 7))
  Downloading gradio-3.33.1-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 39.0 MB/s eta 0:00:00
Requirement already satisfied: markdown in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 8)) (3.5.2)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 9)) (1.23.5)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 10)) (1.5.3)
Collecting Pillow>=9.5.0 (from -r requirements.txt (line 11))
  Downloading pillow-10.2.0-cp310-cp310-manylinux_2_28_x86_64.whl (4.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 60.9 MB/s eta 0:00:00
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 12)) (6.0.1)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 13)) (2.31.0)
Collecting safetensors==0.3.1 (from -r requirements.txt (line 14))
  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 74.5 MB/s eta 0:00:00
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (1.11.4)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 16)) (0.1.99)
Requirement already satisfied: tensorboard in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 17)) (2.15.1)
Collecting transformers==4.31.* (from -r requirements.txt (line 18))
  Downloading transformers-4.31.0-py3-none-any.whl (7.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.4/7.4 MB 72.7 MB/s eta 0:00:00
Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 19)) (4.66.1)
Collecting wandb (from -r requirements.txt (line 20))
  Downloading wandb-0.16.2-py3-none-any.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 74.0 MB/s eta 0:00:00
Collecting bitsandbytes==0.41.0 (from -r requirements.txt (line 22))
  Downloading bitsandbytes-0.41.0-py3-none-any.whl (92.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.6/92.6 MB 9.3 MB/s eta 0:00:00
Collecting llama-cpp-python==0.1.77 (from -r requirements.txt (line 29))
  Downloading llama_cpp_python-0.1.77.tar.gz (1.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 19.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (23.2)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (5.9.5)
Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.21.0->-r requirements.txt (line 1)) (2.1.0+cu121)
Requirement already satisfied: pydantic!=1.7,!=1.7.1,!=1.7.2,!=1.7.3,!=1.8,!=1.8.1,<2.0.0,>=1.6.2 in /usr/local/lib/python3.10/dist-packages (from fastapi==0.95.2->-r requirements.txt (line 5)) (1.10.14)
Collecting starlette<0.28.0,>=0.27.0 (from fastapi==0.95.2->-r requirements.txt (line 5))
  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.0/67.0 kB 8.1 MB/s eta 0:00:00
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio_client==0.2.5->-r requirements.txt (line 6)) (2023.6.0)
Collecting httpx (from gradio_client==0.2.5->-r requirements.txt (line 6))
  Downloading httpx-0.26.0-py3-none-any.whl (75 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 10.1 MB/s eta 0:00:00
Requirement already satisfied: huggingface-hub>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from gradio_client==0.2.5->-r requirements.txt (line 6)) (0.20.3)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from gradio_client==0.2.5->-r requirements.txt (line 6)) (4.5.0)
Collecting websockets (from gradio_client==0.2.5->-r requirements.txt (line 6))
  Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.2/130.2 kB 17.7 MB/s eta 0:00:00
Collecting aiofiles (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (3.9.3)
Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (4.2.2)
Collecting ffmpy (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (3.1.3)
Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (3.0.0)
Requirement already satisfied: markupsafe in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (2.1.4)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (3.7.1)
Collecting mdit-py-plugins<=0.3.3 (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.5/50.5 kB 937.9 kB/s eta 0:00:00
Collecting orjson (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading orjson-3.9.12-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.8/139.8 kB 20.5 MB/s eta 0:00:00
Collecting pydub (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: pygments>=2.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.33.1->-r requirements.txt (line 7)) (2.16.1)
Collecting python-multipart (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.7/45.7 kB 6.8 MB/s eta 0:00:00
Collecting semantic-version (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting uvicorn>=0.14.0 (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading uvicorn-0.27.0.post1-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.7/60.7 kB 9.7 MB/s eta 0:00:00
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.*->-r requirements.txt (line 18)) (3.13.1)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.31.*->-r requirements.txt (line 18)) (2023.12.25)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.31.*->-r requirements.txt (line 18))
  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 55.0 MB/s eta 0:00:00
Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.1.77->-r requirements.txt (line 29)) (5.6.3)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (10.0.1)
Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (0.6)
Collecting dill<0.3.8,>=0.3.0 (from datasets->-r requirements.txt (line 3))
  Downloading dill-0.3.7-py3-none-any.whl (115 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.3/115.3 kB 17.9 MB/s eta 0:00:00
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->-r requirements.txt (line 3)) (3.4.1)
Collecting multiprocess (from datasets->-r requirements.txt (line 3))
  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 19.9 MB/s eta 0:00:00
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 10)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->-r requirements.txt (line 10)) (2023.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 13)) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 13)) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 13)) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->-r requirements.txt (line 13)) (2023.11.17)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (1.4.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (1.60.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (2.17.3)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (1.2.0)
Requirement already satisfied: protobuf<4.24,>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (3.20.3)
Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (67.7.2)
Requirement already satisfied: six>1.9 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (1.16.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard->-r requirements.txt (line 17)) (3.0.1)
Requirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 20)) (8.1.7)
Collecting GitPython!=3.1.29,>=1.0.0 (from wandb->-r requirements.txt (line 20))
  Downloading GitPython-3.1.41-py3-none-any.whl (196 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.4/196.4 kB 26.6 MB/s eta 0:00:00
Collecting sentry-sdk>=1.0.0 (from wandb->-r requirements.txt (line 20))
  Downloading sentry_sdk-1.40.0-py2.py3-none-any.whl (257 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 257.5/257.5 kB 24.9 MB/s eta 0:00:00
Collecting docker-pycreds>=0.4.0 (from wandb->-r requirements.txt (line 20))
  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting setproctitle (from wandb->-r requirements.txt (line 20))
  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)
Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->-r requirements.txt (line 20)) (1.4.4)
Collecting rouge (from auto-gptq==0.3.0+cu117->-r requirements.txt (line 25))
  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (0.4)
Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (4.19.2)
Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (0.12.1)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (6.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.33.1->-r requirements.txt (line 7)) (4.0.3)
Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 20))
  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 10.9 MB/s eta 0:00:00
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 17)) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 17)) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 17)) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 17)) (1.3.1)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 7)) (0.1.2)
Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 7)) (2.0.2)
INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.
Collecting mdit-py-plugins<=0.3.3 (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading mdit_py_plugins-0.3.2-py3-none-any.whl (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.4/50.4 kB 1.4 MB/s eta 0:00:00
  Downloading mdit_py_plugins-0.3.1-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.5/46.5 kB 5.9 MB/s eta 0:00:00
  Downloading mdit_py_plugins-0.3.0-py3-none-any.whl (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.7/43.7 kB 6.7 MB/s eta 0:00:00
  Downloading mdit_py_plugins-0.2.8-py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.0/41.0 kB 6.5 MB/s eta 0:00:00
  Downloading mdit_py_plugins-0.2.7-py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.0/41.0 kB 6.1 MB/s eta 0:00:00
  Downloading mdit_py_plugins-0.2.6-py3-none-any.whl (39 kB)
  Downloading mdit_py_plugins-0.2.5-py3-none-any.whl (39 kB)
INFO: pip is looking at multiple versions of mdit-py-plugins to determine which version is compatible with other requirements. This could take a while.
  Downloading mdit_py_plugins-0.2.4-py3-none-any.whl (39 kB)
  Downloading mdit_py_plugins-0.2.3-py3-none-any.whl (39 kB)
  Downloading mdit_py_plugins-0.2.2-py3-none-any.whl (39 kB)
  Downloading mdit_py_plugins-0.2.1-py3-none-any.whl (38 kB)
  Downloading mdit_py_plugins-0.2.0-py3-none-any.whl (38 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading mdit_py_plugins-0.1.0-py3-none-any.whl (37 kB)
Collecting markdown-it-py[linkify]>=2.0.0 (from gradio==3.33.1->-r requirements.txt (line 7))
  Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 14.7 MB/s eta 0:00:00
  Downloading markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.5/84.5 kB 13.1 MB/s eta 0:00:00
Requirement already satisfied: anyio<5,>=3.4.0 in /usr/local/lib/python3.10/dist-packages (from starlette<0.28.0,>=0.27.0->fastapi==0.95.2->-r requirements.txt (line 5)) (3.7.1)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (1.12)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (3.2.1)
Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (2.1.0)
Collecting h11>=0.8 (from uvicorn>=0.14.0->gradio==3.33.1->-r requirements.txt (line 7))
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 9.5 MB/s eta 0:00:00
Collecting httpcore==1.* (from httpx->gradio_client==0.2.5->-r requirements.txt (line 6))
  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.9/76.9 kB 11.8 MB/s eta 0:00:00
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio_client==0.2.5->-r requirements.txt (line 6)) (1.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 7)) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 7)) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 7)) (4.47.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 7)) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.33.1->-r requirements.txt (line 7)) (3.1.1)
INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.
Collecting multiprocess (from datasets->-r requirements.txt (line 3))
  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.8/134.8 kB 20.8 MB/s eta 0:00:00
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi==0.95.2->-r requirements.txt (line 5)) (1.2.0)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->-r requirements.txt (line 20))
  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (0.33.0)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.33.1->-r requirements.txt (line 7)) (0.17.1)
Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.33.1->-r requirements.txt (line 7)) (1.0.2)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->-r requirements.txt (line 17)) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard->-r requirements.txt (line 17)) (3.2.2)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.10.0->accelerate==0.21.0->-r requirements.txt (line 1)) (1.3.0)
Building wheels for collected packages: llama-cpp-python, peft, ffmpy
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.77-cp310-cp310-linux_x86_64.whl size=276102 sha256=d4a18f25b6865bde63e3fadcc257c17f27f16301b63fba4f43071e3b981074e5
  Stored in directory: /root/.cache/pip/wheels/aa/ed/39/87f2ad350dbbf13b600ac744899186b8647c5323c62e2bb348
  Building wheel for peft (pyproject.toml) ... done
  Created wheel for peft: filename=peft-0.5.0.dev0-py3-none-any.whl size=73123 sha256=9293bbbd4cea4b825fdad6dc5838818a3a3dde4fa31fd96f1e90ab8013f01f74
  Stored in directory: /root/.cache/pip/wheels/ff/57/c1/a023c490307cd8ffa3b61c86c48d9767f0bb850053af18674b
  Building wheel for ffmpy (setup.py) ... done
  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=d29996ef42844f560f617ef83521d97f77515d71424168d7537adaa35754fa33
  Stored in directory: /root/.cache/pip/wheels/01/a6/d1/1c0828c304a4283b2c1639a09ad86f83d7c487ef34c6b4a1bf
Successfully built llama-cpp-python peft ffmpy
Installing collected packages: tokenizers, safetensors, pydub, ffmpy, bitsandbytes, websockets, smmap, setproctitle, sentry-sdk, semantic-version, rouge, python-multipart, Pillow, orjson, markdown-it-py, llama-cpp-python-cuda, llama-cpp-python, h11, einops, docker-pycreds, dill, colorama, aiofiles, uvicorn, starlette, multiprocess, mdit-py-plugins, httpcore, gitdb, transformers, httpx, GitPython, fastapi, exllama, accelerate, wandb, peft, gradio_client, datasets, gradio, auto-gptq
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.15.1
    Uninstalling tokenizers-0.15.1:
      Successfully uninstalled tokenizers-0.15.1
  Attempting uninstall: safetensors
    Found existing installation: safetensors 0.4.2
    Uninstalling safetensors-0.4.2:
      Successfully uninstalled safetensors-0.4.2
  Attempting uninstall: Pillow
    Found existing installation: Pillow 9.4.0
    Uninstalling Pillow-9.4.0:
      Successfully uninstalled Pillow-9.4.0
  Attempting uninstall: markdown-it-py
    Found existing installation: markdown-it-py 3.0.0
    Uninstalling markdown-it-py-3.0.0:
      Successfully uninstalled markdown-it-py-3.0.0
  Attempting uninstall: mdit-py-plugins
    Found existing installation: mdit-py-plugins 0.4.0
    Uninstalling mdit-py-plugins-0.4.0:
      Successfully uninstalled mdit-py-plugins-0.4.0
  Attempting uninstall: transformers
    Found existing installation: transformers 4.35.2
    Uninstalling transformers-4.35.2:
      Successfully uninstalled transformers-4.35.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
lida 0.0.10 requires kaleido, which is not installed.
imageio 2.31.6 requires pillow<10.1.0,>=8.3.2, but you have pillow 10.2.0 which is incompatible.
Successfully installed GitPython-3.1.41 Pillow-10.2.0 accelerate-0.21.0 aiofiles-23.2.1 auto-gptq-0.3.0+cu117 bitsandbytes-0.41.0 colorama-0.4.6 datasets-2.16.1 dill-0.3.7 docker-pycreds-0.4.0 einops-0.7.0 exllama-0.0.9+cu117 fastapi-0.95.2 ffmpy-0.3.1 gitdb-4.0.11 gradio-3.33.1 gradio_client-0.2.5 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 llama-cpp-python-0.1.77 llama-cpp-python-cuda-0.1.77+cu117 markdown-it-py-2.2.0 mdit-py-plugins-0.3.3 multiprocess-0.70.15 orjson-3.9.12 peft-0.5.0.dev0 pydub-0.25.1 python-multipart-0.0.6 rouge-1.0.1 safetensors-0.3.1 semantic-version-2.10.0 sentry-sdk-1.40.0 setproctitle-1.3.3 smmap-5.0.1 starlette-0.27.0 tokenizers-0.13.3 transformers-4.31.0 uvicorn-0.27.0.post1 wandb-0.16.2 websockets-12.0
WARNING: The following packages were previously imported in this runtime:
  [PIL]
You must restart the runtime in order to use newly installed versions.
Collecting gradio==3.32.0
  Downloading gradio-3.32.0-py3-none-any.whl (19.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/19.9 MB 16.1 MB/s eta 0:00:00
Requirement already satisfied: aiofiles in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (23.2.1)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.9.3)
Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (4.2.2)
Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.95.2)
Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.3.1)
Requirement already satisfied: gradio-client>=0.2.4 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.2.5)
Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.26.0)
Requirement already satisfied: huggingface-hub>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.20.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.1.3)
Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.2.0)
Requirement already satisfied: markupsafe in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.1.4)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.7.1)
Requirement already satisfied: mdit-py-plugins<=0.3.3 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.3.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.23.5)
Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (3.9.12)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.5.3)
Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (10.2.0)
Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (1.10.14)
Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.25.1)
Requirement already satisfied: pygments>=2.12.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.16.1)
Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.0.6)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (6.0.1)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.31.0)
Requirement already satisfied: semantic-version in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (2.10.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (4.5.0)
Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (0.27.0.post1)
Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio==3.32.0) (12.0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (0.4)
Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (4.19.2)
Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio==3.32.0) (0.12.1)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.2.4->gradio==3.32.0) (2023.6.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.2.4->gradio==3.32.0) (23.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio==3.32.0) (3.13.1)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio==3.32.0) (4.66.1)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (0.1.2)
Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (2.0.2)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio==3.32.0) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio==3.32.0) (2023.4)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.32.0) (8.1.7)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn>=0.14.0->gradio==3.32.0) (0.14.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (6.0.4)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio==3.32.0) (4.0.3)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio==3.32.0) (0.27.0)
Requirement already satisfied: anyio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (3.7.1)
Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (2023.11.17)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (1.0.2)
Requirement already satisfied: idna in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (3.6)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio==3.32.0) (1.3.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (4.47.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (1.4.5)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio==3.32.0) (3.1.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->gradio==3.32.0) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->gradio==3.32.0) (2.0.7)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.32.0) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.32.0) (0.33.0)
Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.32.0) (0.17.1)
Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.32.0) (1.0.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->gradio==3.32.0) (1.16.0)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio->httpx->gradio==3.32.0) (1.2.0)
Installing collected packages: gradio
  Attempting uninstall: gradio
    Found existing installation: gradio 3.33.1
    Uninstalling gradio-3.33.1:
      Successfully uninstalled gradio-3.33.1
Successfully installed gradio-3.32.0
Found existing installation: llama_cpp_python 0.1.77
Uninstalling llama_cpp_python-0.1.77:
  Successfully uninstalled llama_cpp_python-0.1.77
Collecting llama-cpp-python==0.1.78
  Downloading llama_cpp_python-0.1.78.tar.gz (1.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 9.9 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.1.78) (4.5.0)
Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.1.78) (1.23.5)
Requirement already satisfied: diskcache>=5.6.1 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python==0.1.78) (5.6.3)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.78-cp310-cp310-linux_x86_64.whl size=5811091 sha256=209d670a27a6691b40380caee617af17d99b896bfef56b64fabfce5f603fd2db
  Stored in directory: /tmp/pip-ephem-wheel-cache-rhs2coaq/wheels/61/f9/20/9ca660a9d3f2a47e44217059409478865948b5c8a1cba70030
Successfully built llama-cpp-python
Installing collected packages: llama-cpp-python
Successfully installed llama-cpp-python-0.1.78

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
f1cf97|OK  |   164MiB/s|/content/text-generation-webui/models//wizardlm-1.0-uncensored-llama2-13b.ggmlv3.q5_K_M.bin

Status Legend:
(OK):download completed.
/content/text-generation-webui
2024-02-03 08:22:25 WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
2024-02-03 08:22:29.859509: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-03 08:22:29.859583: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-03 08:22:29.861007: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-03 08:22:31.839727: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-02-03 08:22:33 INFO:Loading wizardlm-1.0-uncensored-llama2-13b.ggmlv3.q5_K_M.bin...
CUDA error 35 at /tmp/pip-install-nydaa5se/llama-cpp-python_720d0f2868d94d788fd2ec34324b390a/vendor/llama.cpp/ggml-cuda.cu:4883: CUDA driver version is insufficient for CUDA runtime version

Error executing "wizard-vicuna-13B.ggml.q4_0 (using llama.cpp)" on Colab

Hi, I am getting this error whentriying to execute "wizard-vicuna-13B.ggml.q4_0 (using llama.cpp)" :

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.
/content/text-generation-webui
2023-10-05 19:35:57 WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-10-05 19:36:00.774040: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
2023-10-05 19:36:05 INFO:Loading wizard-vicuna-13B-GPTQ...
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/text-generation-webui/server.py:1017 in <module>                    │
│                                                                              │
│   1014 │   │   update_model_parameters(model_settings, initial=True)  # hija │
│   1015 │   │                                                                 │
│   1016 │   │   # Load the model                                              │
│ ❱ 1017 │   │   shared.model, shared.tokenizer = load_model(shared.model_name │
│   1018 │   │   if shared.args.lora:                                          │
│   1019 │   │   │   add_lora_to_model(shared.args.lora)                       │
│   1020                                                                       │
│                                                                              │
│ /content/text-generation-webui/modules/models.py:74 in load_model            │
│                                                                              │
│    71 │   │   │   │   return None, None                                      │
│    72 │                                                                      │
│    73 │   shared.args.loader = loader                                        │
│ ❱  74 │   output = load_func_map[loader](model_name)                         │
│    75 │   if type(output) is tuple:                                          │
│    76 │   │   model, tokenizer = output                                      │
│    77 │   else:                                                              │
│                                                                              │
│ /content/text-generation-webui/modules/models.py:293 in ExLlama_HF_loader    │
│                                                                              │
│   290 def ExLlama_HF_loader(model_name):                                     │
│   291 │   from modules.exllama_hf import ExllamaHF                           │
│   292 │                                                                      │
│ ❱ 293 │   return ExllamaHF.from_pretrained(model_name)                       │
│   294                                                                        │
│   295                                                                        │
│   296 def get_max_memory_dict():                                             │
│                                                                              │
│ /content/text-generation-webui/modules/exllama_hf.py:92 in from_pretrained   │
│                                                                              │
│    89 │   │   │   if len(found) > 0:                                         │
│    90 │   │   │   │   weight_path = found[-1]                                │
│    91 │   │   │   │   break                                                  │
│ ❱  92 │   │   assert weight_path is not None, f'could not find weight in "{p │
│    93 │   │                                                                  │
│    94 │   │   config.model_path = str(weight_path)                           │
│    95 │   │   config.max_seq_len = shared.args.max_seq_len                   │
╰──────────────────────────────────────────────────────────────────────────────╯
AssertionError: could not find weight in "models/wizard-vicuna-13B-GPTQ"

ImportError: libcudart.so.12

when I tried to run Mistral-7B-OpenOrca (using oobabooga/text-generation-webui)Ж
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

Integrate with LiteLLM - Evaluate 100+LLMs, 92% faster

Hi @Troyanovsky @klipski I'm the maintainer of LiteLLM. we allow you to create a proxy server to call 100+ LLMs to make it easier to run benchmark / evals .

I'm making this issue because I believe LiteLLM makes it easier for you to run benchmarks and evaluate LLMs (I'd love your feedback if it does not)

Try it here: https://docs.litellm.ai/docs/simple_proxy
https://github.com/BerriAI/litellm

Using LiteLLM Proxy Server

Creating a proxy server

Ollama models

$ litellm --model ollama/llama2 --api_base http://localhost:11434

Hugging Face Models

$ export HUGGINGFACE_API_KEY=my-api-key #[OPTIONAL]
$ litellm --model claude-instant-1

Anthropic

$ export ANTHROPIC_API_KEY=my-api-key
$ litellm --model claude-instant-1

Palm

$ export PALM_API_KEY=my-palm-key
$ litellm --model palm/chat-bison

Using to run an eval on lm harness:

python3 -m lm_eval \
  --model openai-completions \
  --model_args engine=davinci \
  --task crows_pairs_english_age

llava 13b

Have you tested llava 13b v1.5 yet ?

Which runs on least powerful hardware...

Do you have any data/sorting of which ones used the least amount of resources? Versus which did the best. The usecase I have in mind is in a very constrained server environment, so something that doesn't use much memory or CPU would be a key driver in picking one, versus which does best. I'm thinking in the 2 GB of ram space....

Legal

Fantastic work!
Do you know of models or LORA trained on US law? Sort of lawyer LLM.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.