I attempted to load the Mistral Open Orca model, but I couldn't, so I changed the code a little, but now I receive an answer if I ask the same query again, so the first time it gives no answer, and the second time it responds, maybe from the cache. This is all new to me.

Please let me know if there is a method to import the "mistral-7b-openorca.Q4_K_M.gguf" model without editing.

My edit:

def build_llm(model):
    # Call the function to get the number of threads to use
    num_threads = determine_threads_to_use()
    print(f"Number of Threads avaialble : {num_threads}")

    llm = LlamaCpp(
        model_path=cfg.MODEL_BIN_DIR + "/" + model,
        n_gpu_layers=num_threads,
        # f16_kv=True,
        n_batch=8192 / 4,
        n_ctx=8192,
        max_tokens=cfg.MAX_NEW_TOKENS,
        n_threads=num_threads,
        temperature=cfg.TEMPERATURE,
        streaming=True,
        repeat_penalty= 1.3,
        callbacks=[StreamingStdOutCallbackHandler()],
    )

    # Local CTransformers model
    # llm = CTransformers(
    #     model=cfg.MODEL_BIN_DIR + "/" + model,
    #     model_type="llama",
    #     config={
    #         "max_new_tokens": cfg.MAX_NEW_TOKENS,
    #         "temperature": cfg.TEMPERATURE,
    #         "threads": num_threads,
    #         "stream": True,
    #         "repetition_penalty": 1.3,
    #     },
    #     callbacks=[StreamingStdOutCallbackHandler()],
    # )


    return llm

If I use the default code, the error is:

RuntimeError: Failed to create LLM 'llama' from 'models/mistral-7b-openorca.Q4_K_M.gguf'.

No matter what model type I change.

huggingface.py", line 85, in
texts = list(map(lambda x: x.replace("\n", " "), texts))
^^^^^^^^^
AttributeError: 'Message' object has no attribute 'replace'

hello, I am trying to use your code however I encounter this error:

Error in __cdecl faiss::FileIOReader::FileIOReader(const char *) at D:\a\faiss-wheels\faiss-wheels\faiss\faiss\impl\io.cpp:68: Error: 'f' failed: could not open vectorstore\db_faiss\index.faiss for reading: No such file or directory

Have you encountered it and if so how to resolve it

Thank you

Hi
after run got this error:

chainlit run main.py -w
Traceback (most recent call last):
File "/usr/local/bin/chainlit", line 8, in
sys.exit(cli())
^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/chainlit/cli/init.py", line 152, in chainlit_run
run_chainlit(target)
File "/usr/local/lib/python3.11/site-packages/chainlit/cli/init.py", line 45, in run_chainlit
load_module(config.run.module_name)
File "/usr/local/lib/python3.11/site-packages/chainlit/config.py", line 244, in load_module
spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in _call_with_frames_removed
File "/opt/PrivateDocBot/main.py", line 11, in
init_vector_db()
File "/opt/PrivateDocBot/src/loader.py", line 50, in init_vector_db
files = [file for file in os.listdir(cfg.DB_FAISS_PATH) if file != 'readme.md']
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'vectorstore\db_faiss'

Any idea?
Thanks