Can promptify support exporting formatted matched terms with identifiers like PubTator or BERN2:
Example:

1. https://www.ncbi.nlm.nih.gov/research/pubtator-api/publications/export/pubtator?pmids=28483577
2. http://bern2.korea.ac.kr/

Thanks,

hh1985

Both of these example templates result in the error TemplateSyntaxError: unexpected char '\\' at 5:

ner.jinja:

{{ ' "hello world" ' }}

ner.jinja:

{{ " \"\"\"hello world\"\"\" " }}

And I confirmed it's not an issue with jinja, because passing the same template directly to jinja works fine:

>>> jinja2.Template('{{ \' "hello world" \' }}').render()
' "hello world" '

>>> jinja2.Template('{{ " \\"\\"\\"hello world\\"\\"\\" " }}\n').render()
' """hello world""" '

First I install this promptify using the below command

pip3 install git+https://github.com/promptslab/Promptify.git

Then, I want to use the relation extraction model by the below code

 result = nlp_prompter.fit('relation_extraction.jinja',
                              domain='clinical',
                              text_input=sentence)

However, I met this error 'jinja2.exceptions.TemplateNotFound: relation_extraction.jinja'

I debuged and found that there is really no such model， as showed in the below figure.

What I should do to solve this problem. Thank you!

As mentioned in the code I executed the code and downloaded everything from requirements.txt other codes are working but

Code for Relation Extraction

result= nlp_prompter.fit('relation_extraction.jinja', 
                                   domain = 'clinical',
                                   # examples = [prompt_examples[0]],
                                   text_input  = sent)
print(result['text'])

gives the error "TemplateNotFound". What should I do?

The content of the following metadata files is empty :

• promptify/prompts/text2text/qa/metadata.json
• promptify/prompts/text2text/qa_gen/metadata.json

This bug causes this exception to be thrown.

Issue was in Question-Generation prompt, Need to check and fix it.

Issue code:

from promptify import OpenAI
from promptify import Prompter

context = "The patient is a 93-year-old female with a medical history of chronic right hip pain, osteoporosis, hypertension, depression, and chronic atrial fibrillation admitted for evaluation and management of severe nausea and vomiting and urinary tract infection"

result = nlp_prompter.fit('qa_gen.jinja',
                                      text_input=context,
                                      domain="medical",
                                      total_questions=5,
                                      max_QA_tokens=10
                                     )

The output is :

[{'Q': 'What is the patient's age?', 'A': '93 years old'}, 
{'Q': 'What medical conditions does the patient have?', 'A': 'Chronic right hip pain, osteoporosis, hypertension, depression, and chronic atrial fibrillation'}, 
{'Q': 'What is the patient admitted for?', 'A': 'Evaluation and management of severe nausea and vomiting and urinary tract infection'}, 
{'Q': 'What is the patient's gender?', 'A': 'Female'}, 
{'Q': 'What is the patient's primary complaint?', 'A': 'Severe nausea and vomiting'}]

Parsing this output with eval() gives error because of single and double quote issue.

I tried installing promptify in my local i get the following error "ModuleNotFoundError: No module named 'promptify.models.nlp'
installed promptify as well and other relevent packages but no luck.This is the case as well with colab notebook provided please find the link below. can anyone help.

https://colab.research.google.com/drive/16DUUV72oQPxaZdGMH9xH1WbHYu6Jqk9Q?usp=sharing

I am trying to follow the examples for using promptify with huggingface, but I run into this error trying to run the code:

library was installed directly from github and everything from requirements.txt is installed

collab notebook with reproduceable error

Hi,

It would be great if you could add more comprehensive test cases and examples so that the models can be more accurately evaluated.

Moreover, there is no explanation about the concept of jinja, which has been frequently used in your examples.

Can it support Chinese? Want to use this for Chinese named entity extraction

Facing this error, from the opensai_model.py file. This error sometimes comes and sometimes everything runs smooth

Adding text input to SQL code conversion feature

Add ToolFormer

Toolformer: Language Models Can Teach Themselves to Use Tools
https://arxiv.org/abs/2302.04761

Contributions & PRs are welcome

• sentiment analysis
• named entity recognition
• text classification
• topic modeling
• summarization
• text generation
• text normalization
• text segmentation
• information extraction
• keyphrase extraction
• paraphrasing
• Grammar correction
• text clustering
• text similarity
• text coherence
• text alignment
• text annotation
• word sense disambiguation
• part-of-speech tagging
• chunking
• dependency parsing
• constituency parsing
• semantic role labeling
• coreference resolution
• text-to-image generation
• text-to-video generation
• question answering
• text comprehension
• dialogue generation
• dialogue management
• one format to another format conversion
• Outliers detection
• translation
• code comments & docstring
• emotion recognition
• sarcasm detection
• irony detection
• humor recognition
• opinion mining
• stance detection
• argumentation mining
• semantic search
• question generation
• text editing
• data cleaning
• data preprocessing
• data normalization
• data integration
• data wrangling
• data analysis
• data interpretation
• data synthesis
• data masking
• data anonymization

Is there any enhancement plan to include code davinci models?

I run the follow code

import json

relation_examples = json.load(open("examples/data/relation_extraction.json",'r'))
print("Got", len(relation_examples), "examples.")

prompt_examples = []
for sample in relation_examples:
    prompt_examples.append((sample['text'], sample['labels']))
print()

then I met this error "No such file or directory: 'examples/data/relation_extraction.json'".

I checked your code, I can't find this file.
can you give one example for relation_extraction.json?

Thank you!

With python3.10, the code fails with ImportError: cannot import name 'soft_unicode' from 'markupsafe'.

aws/aws-sam-cli#3661 (comment) helps fixing that.
Still with that change, with the introductory example we get

    assert len(variables_missing) == 0, f"Missing required variables in template {variables_missing}"
AssertionError: Missing required variables in template ['labels']

It would be great if the main setup instructions work out of the box.

source data format = 'txt'
destination data format = 'json'

This is an exciting model. How can I add few examples instead of one shot?
I am trying to identify brand name from ecommerce product titles such as Amazon.

For example:
One shot : Among US Hat Cap Snapback Gift US Seller
BRAND, Among US
Two shot: [New Adhesion Series 5] Braun Mens Electric Shaver 50-R1200s
BRAND, Braun

Any way I can do this?

thanks

Branch - large-refactoring
https://github.com/promptslab/Promptify/tree/large-refactoring

• Base model with rate error and retry handling
• Mock Model
• Test Cases for Base Model using Mock model
• Improved Openai-Completion Model class
• A robust parser class to handle incomplete JSON outputs from LLMs
• Test cases for Openai-Completion Model class
• Improved Openai-Chat Model class
• Test cases for Openai-Chat Model class
• Improved Hugginface Model class
• Test cases for the Hugginface Model class
• Antrophic model class
• Test cases for the Antrophic model class
• Co:here model class
• Test cases for the Co:here model class
• Prompt-Processor (Batch prompt etc.)
• Improved Prompter class
• Improve templates (such as adding keyword position in ner.jinja etc)
• generate data and train hf model
• add examples for different use-cases
• proper doc for functions and examples

I create xxx.jinja as my own templates, then I tried theses code:
prompt = nlp_prompter.generate_prompt('business_analysis.jinja', text_input="GMV下降的原因", domain="business analysis" ) print(prompt) sentence="GMV下降的原因" result = nlp_prompter.fit('business_analysis.jinja',domain='business analysis',text_input=sentence, labels=None) print(result)

here is the question, the "generate_prompt" and "fit" function can't find this template, I tried the absolute path, and even put this template along with the templates of Promptify

Could you please do me a favor

Add local model support in /models/inference/ to work with ner, question-answering etc prompts

Hi,

I'm trying to run the example in the README, but I'm getting an error.

The source code I used was this:

from promptify import OpenAI
from promptify import Prompter

sentence     =  """The patient is a 93-year-old female with a medical
                history of chronic right hip pain, osteoporosis,
                hypertension, depression, and chronic atrial
                fibrillation admitted for evaluation and management
                of severe nausea and vomiting and urinary tract
                infection"""

model        = OpenAI("sk-MYKEY")
nlp_prompter = Prompter(model)


result       = nlp_prompter.fit('ner.jinja', domain ='medical', text_input  = sentence)
print(result)

And the error that occurs is this:

File "./teste.py", line 15, in <module>
    result       = nlp_prompter.fit('ner.jinja', domain ='medical', text_input  = sentence)
  File "/home/abner/Devel/daschat/openai/promptify/.venv/lib/python3.8/site-packages/promptify/prompts/nlp/prompter.py", line 52, in fit
    prompt = self.generate_prompt(template_name, **prompt_kwargs)
  File "/home/abner/Devel/daschat/openai/promptify/.venv/lib/python3.8/site-packages/promptify/prompts/nlp/prompter.py", line 38, in generate_prompt
    assert len(variables_missing) == 0, f"Missing required variables in template {variables_missing}"
AssertionError: Missing required variables in template ['labels']

Any idea what I'm doing wrong?

Thanks.

Using GPT to create apis based on promptify

I want to classify my text into multiclass with confidence percentage and want this following result into json format only.

Can someone help in editing the repo specific files.

I am using promptify for solving NER Problem everything works fine except sometimes when it returns an Error for Nonetype which means it's empty but on the exception I am also passing it as a plain string but still same error.
It works fine after pausing for 1 minute or more but again it returns NoneType when I am testing it for continous requests.

from promptify.models.nlp.openai_model import OpenAI
from promptify.prompts.nlp.prompter import Prompter


try:
    model = OpenAI(openai.api_key)
except Exception as e:
    print("Error",e)
    model = OpenAI("me-2asdasdasdT123123.........")

#Initialize Prompter
nlp_prompter = Prompter(model)
ner_result = nlp_prompter.fit('ner.jinja',
                        domain      = 'Hospitality',
                        text_input=str('My name is Aymal'),
                        description= "The API response below contains user name"
                        )
 print("NER RESULT",ner_result)

Using ARM processor if that matters:

(base) root@ip-172-31-38-54:/home/ubuntu# python
Python 3.7.6 | packaged by conda-forge | (default, Mar 23 2020, 22:25:07)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from promptify import OpenAI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniforge3/lib/python3.7/site-packages/promptify/__init__.py", line 2, in <module>
    from .models.nlp.openai_model import OpenAI
  File "/root/miniforge3/lib/python3.7/site-packages/promptify/models/nlp/openai_model.py", line 6, in <module>
    from .utils import get_encoder
  File "/root/miniforge3/lib/python3.7/site-packages/promptify/models/nlp/utils/__init__.py", line 1, in <module>
    from .bpe_encoder import get_encoder
  File "/root/miniforge3/lib/python3.7/site-packages/promptify/models/nlp/utils/bpe_encoder.py", line 8, in <module>
    from .utils import download
  File "/root/miniforge3/lib/python3.7/site-packages/promptify/models/nlp/utils/utils.py", line 34
    if last_modified := response.headers.get("last-modified"):
                      ^
SyntaxError: invalid syntax
>>>

https://platform.openai.com/docs/guides/chat

Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from promptify import OpenAI
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\pigna\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\promptify\__init__.py", line 1, in <module>
    from .models.nlp.hub_model import HubModel
  File "C:\Users\pigna\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\promptify\models\nlp\hub_model.py", line 6, in <module>
    from promptify import __version__
ImportError: cannot import name '__version__' from partially initialized module 'promptify' (most likely due to a circular import) (C:\Users\pigna\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\promptify\__init__.py)

I run this to install python3 -m pip install git+https://github.com/promptslab/Promptify.git

Saw NER notebook. Is it possible to see the prompt it would have generated behind the seen, especially with the few shots example with domain knowledge?

• Custom template support,
• Azure platform support,
• GPT-3.5 support with more features
• Word sense disambiguation,
• Hugginface local model support
• Prompt optimization features,
• Additional templates (such as trip planning templates),
• Better documentation with examples,
• Image embeddings and image model support,
• Prompt generator supports based on different parameters.
• cost calculation for the prompt and optimization suggestions

Currently, these are the default model parameters if the user doesn't specify them

        model_name: str = "text-davinci-003",
        temperature: float = 0.7,
        max_tokens: int = 4000,
        top_p: float = 0.1,
        frequency_penalty: float = 0,
        presence_penalty: float = 0,
        stop: Union[str, None] = None,

But in practice, the typical parameters can be quite different for different tasks. For example, we would typically want temperature = 0 for highly technical tasks like classification or QA. Meanwhile, we typically want penalties to be 0 for most use cases, except for tasks like summarization, where we typically want nonzero penalties.

It would greatly improve out-of-the-box accuracy if we specified different default parameters for each individual template.

Hi,
Thank you for such great work.

Could you please add huggingface models or any other available models on the web to this package so that we can test it with open-source models?

14/7/2023 : successfully runing

`examples = [list of dictionnaries here as examples]
data = list_of_product_names

result = nlp_prompter.fit('ner.jinja',
domain = 'ecommerce',
text_input = f'{data}',
labels = ["NAME", "WEIGHT","VOLUME","COUNT"],
examples = examples)`

after the update of 17/7/2023 :

`examples = [list of dictionnaries here as examples]

data = list_of_product_names

model = OpenAI(api_key) # or HubModel() for Huggingface-based inference or 'Azure' etc
prompter = Prompter('ner.jinja') # select a template or provide custom template
pipe = Pipeline(prompter , model)

result = pipe.fit(domain = 'ecommerce',
text_input = new_input,
labels = ["NAME", "WEIGHT","VOLUME","COUNT"],
examples = examples )`

the code block keeps running forever with no results, even with a short sentence.
when I take off examples it runs faster but it doesn't not yeild the desired output format.

Could you please add documentation for examples, benchmarks, and API? It is somehow complicated to know the parameters of each model and function.

Hello @monk1337 ,
Can you kindly have a look? I used the code from the documentation.

Hi @monk1337,

I guess something in text_input parameter of nlp_prompter changed after the recent updates you made.
Check the attached screenshot, which I tried on your few shot example.

The response for the NER template is a string representation of a list of dictionaries.

Currently this means the user first needs to parse and convert into list of dicts to use the output effectively.

Hello !
Can we add more few shots when we are doing a request?

Right know I have a list with 1000 examples and when I;m trying to do this I received an error that max tokens were exceed.

How can we use more shots?

Thanks,
Alex

It would bring a lot of value to the community to have this feature. Are you concidering it's implementation?

Hey, I saw Image Data task is introduced to be implemented at the bottom of the readme, I wonder what is the context here, if I understand what that is then I can try to draft a PR if I can.

Currently, a selection of few models is available. Will you be adding new open-source models like LLaMA 2 and Falcon?

ChatGPT usually has the latest GPT version. Can Promptify integrate reverse engineered ChatGPT api solution like https://github.com/acheong08/ChatGPT.

Is it possible to bump or loosen up the jinja2 version to support 3.x.x from the current jinja2==2.11.3? I'm having version conflicts with some packages I'm using like prefect. Thank you.

This is the network call which happens while doing the NER.

'Request to OpenAI API' method=post path=https://api.openai.com/v1/completions
data='{"model": "text-davinci-003", "prompt": "It is related to detection of symptoms and diseases for medical domain\nYou are a highly intelligent and accurate medical domain Named-entity recognition(NER) system. You take Passage as input and your task is to recognize and extract specific types of medical domain named entities in that given passage and classify into a set of following predefined entity types:\n['SYMPTOM', 'DISEASE']Your output format is only [{{'T': type of entity from predefined entity types, 'E': entity in the input text}},...,{{'branch' : Appropriate branch of the passage ,'group': Appropriate Group of the passage}}] form, no other form.\n\nExamples:\n\nInput: Leptomeningeal metastases (LM) occur in patients with breast cancer (BC) and lung cancer (LC). The cerebrospinal fluid (CSF) tumour microenvironment (TME) of LM patients is not well defined at a single-cell level. We did an analysis based on single-cell RNA sequencing (scRNA-seq) data and four patient-derived CSF samples of idiopathic intracranial hypertension (IIH)\nOutput: [[{'E': 'DISEASE', 'W': 'Leptomeningeal metastases'}, {'E': 'DISEASE', 'W': 'breast cancer'}, {'E': 'DISEASE', 'W': 'lung cancer'}, {'E': 'BIOMARKER', 'W': 'cerebrospinal fluid'}, {'E': 'DISEASE', 'W': 'tumour microenvironment'}, {'E': 'TEST', 'W': 'single-cell RNA sequencing'}, {'E': 'DISEASE', 'W': 'idiopathic intracranial hypertension'}]]\n\nInput: The patient is a 93-year-old female with a medical history of chronic right hip pain, osteoporosis, hypertension, depression, and chronic atrial fibrillation admitted for evaluation and management of severe nausea and vomiting and urinary tract infection\nOutput:", "temperature": 0.7, "max_tokens": 3550, "top_p": 0.1, "frequency_penalty": 0, "presence_penalty": 0, "stop": null}'

We are asking the output to be in the format => {'T': type of entity from predefined entity types, 'E': entity in the input text. }

However, in the one shot examples we provide it in the format E: Entity Type, W: Entity name.

Isn't this incorrect? If it isn't what is the reason for providing the examples in this format?

Hello ,
i was wondering if its possible to use Promptfy to return the NER and the index of the detected entities , that will help alot in NER tasks ( cause some text have multipe NERs and having the index will help bypassing that issue