nicolay-r / deep-book-processing Goto Github PK

https://github.com/nicolay-r/chatbot_experiments/blob/4cd8212e155afd259f849c3d24e17cc5633af4f1/my_s4_char_spectrum_embed.py#L47

Main branch: https://github.com/nicolay-r/chatbot_experiments/tree/dataset-v3.1

• Adopt candidates selection from neg for spectrum based version #15
• Select the grammatically close sentences in candidates from neg for the spectrum version (requires sentence embedding application) (done in c51d66f)
  

Result dataset stat

List of the modeling of personalities.

• SPECTRUM
  • occurance of the whole adjective words (no need to substract rich/poor).
    -- SPECTRUM bag-of-words
  • is what i've done.
    • rich -- embedding.
    • poor -- embedding.
      -- how many words occurst in context for character.
      -- SPECTRUM embedding
• HLA Dictionary

Refactoring: Folds should have unique speakers

• Select candidates from the related folding index (ab118d9)
• Check absense of intersections with mentioned speakers (822b54b)

Explanation: it is a bug, because I did not expect to see IDs in utterances and prevent from the following overfitting.
However, it would be great to not to consider name of others!
So, solution is to completely mask names
#12 related

• #28
• #18
• #27
  
• #26
• Switch to 5 folding parts instead of 10
  
• #25
  
• #29
• #30
• #35
• Fixed to Persona: none in the case when persona is absent (df6bfd0)
• #36
• oversampled candidates are the same [bug] (04b18fa)
• fixed bug with absense of the newline after a set of oversampled examples (c8f400b)
• spectrums are not randomly selected (41addd9)

• analyse original amount of books and speakers (e5209cb)

Mainly need to follow the code of the ALOHA paper related to negative candidate responses selection:

1. This is a code for selecting candidates:
   https://github.com/newpro/aloha-chatbot/blob/c516e717c2af6fceff86465a44705ed8ff044387/src/core.py#L139-L186
2. It requres the pre-trained matrix factorization:
   https://github.com/newpro/aloha-chatbot/blob/c516e717c2af6fceff86465a44705ed8ff044387/src/matrix_factorization.py#L32

MatrixWrapper(hla_d, col1='char_id', col2='feature')

3. All the related code is wrapped into write dataset method:
   https://github.com/newpro/aloha-chatbot/blob/c516e717c2af6fceff86465a44705ed8ff044387/src/core.py#L205C9-L205C14

they use implicit python library for model training

For the factorization we need the dataset of features.

Here, is the link where the related dataset could be for the ALOHA paper:
https://cs.uwaterloo.ca/~jhoey/research/aloha/

This is how their features-related dataset looks like:

How to retreive the factors:

model.user_factors

#13 related

We need to provide oversamplng with features and taking parts of the utterances in dialogs.

• Oversampling is applicable only for training (ab118d9)

https://github.com/nicolay-r/chatbot_experiments/blob/9d37f2c0c87888ac7cff820d0ea1c7ebacbff85d/utils_my.py#L189-L206

• keep 40 most representative traits for further selection of 8 out of them (b03054b)
• fix api for saving/loading spectrums (d19bbab)
• add distrubution based random selection (464082b)

There is a way to experiment straightaway in ParlAI:

https://parl.ai/docs/agent_refs/hugging_face.html#gpt2

!parlai train_model -m hugging_face/gpt2 --add-special-tokens True 
  --add-start-token True --gpt2-size medium -t gutenbergbookchars -bs 6 
  -mf parlai/gpt-2 -veps 0.5 --tensorboard_log True

This is a main algorithm:

Prefix annotation

We consider text comments analysis for the presence of speakers and their relation with the predecessed utterance. In particular, in this work we aimed on prefixes in between the: 1. Utterance 2. Speaker in comment. Prefixes represent a subject of analysis that results in lexicon construction. The result lexicon is neccessary in utterances annotation.

To reveal the connection between character and utterance in comment, we follow the assumption of the relatively short distance in between (prefix).
Infact, comments with prefixes which length in terms 1, 2 and 3 were considered.

Given a whole set of books $B$. First step, we collect the whole dialogs $d_i$ for every book $i \in B$. The result is a set $D$, where $d_i \in D$ for each $i \in B$

Next step we analysie filtered comments. We follow the assumption that authors of the different books has the similar terms in comentary the related character.
Therefore, the goal of the related step is to reveal the most representative terms in comments.

To provide the related assessments, the tf-idf measure was chosen.

In particular, we calculate: tf-idf for every term (tfa is considered).
Finally we use the:

1. p_threshold to filter the most representative terms for lexicon.

https://github.com/nicolay-r/chatbot_experiments/blob/731d1a1892d36e155829e1fd9f095250636ec7a2/ceb_books_2_utterance_prefixes.py#L8-L49

2. Trigrams ended by to and at (utterances directed to other character)
3. Rule 2 is actually becomes an exceptional in case when such prefixes is the only terms between utterance and caharacter.
   Corner cases filtering:
   https://github.com/nicolay-r/chatbot_experiments/blob/731d1a1892d36e155829e1fd9f095250636ec7a2/ceb_books_2_utterance_prefixes.py#L52-L65

Matching Algorith (Establishing Connection)

We consider:

• characters that mentioned straightaway after the utterance in comment (absece of prefix)
• patter-matching of the related prefix, using the constructed lexicon

https://github.com/nicolay-r/chatbot_experiments/blob/3b83e2e8730a6d3d5b59e82671001ba135598dc9/core/speaker_annotation.py#L7-L30

Conclusion

This part might become a source of the further optimization and improvements, applcation of LLM.
It could be actually fact-checked with LLM.

Explanation: not only from the same book, version 3.0

#34 related

We consider third-party resouce, contains crowd-sourced personality traits of 800 fictional characters from numerous fictional works. Each character is scored on 264 adjective-pair spectrums, such as "trusting / suspicious" or "luddite / technophile".

https://github.com/tacookson/data/blob/master/fictional-character-personalities/personalities.txt

We consider spectrum_low and spectrum_high parameter from these values.

There are two of them:

1. Paragraph based (N = 1)
   https://github.com/nicolay-r/chatbot_experiments/blob/3b83e2e8730a6d3d5b59e82671001ba135598dc9/core/book/utils.py#L10-L47

2. Comments-based (text part in between seegments of the dialog utterances)
   https://github.com/nicolay-r/chatbot_experiments/blob/3b83e2e8730a6d3d5b59e82671001ba135598dc9/core/utils_comments.py#L5-L19

• speaker utterances to 100
• folding parts to 10
• #8 (not in this version)
• utterances in replies one speaker (should not have leverages); consider limit to 100

This and related parameters should guarantee that speakers are separated into words / terms
https://github.com/nicolay-r/chatbot_experiments/blob/55bdb9df98018c06b7c4bbf614e07a97c14a31dd/test/my_s4_1_char_spectrum_cat_paragraphs.py#L25-L27

Fix proposal: 9f10e29

https://github.com/nicolay-r/chatbot_experiments/blob/520c115ccd4611110d82b45956434e8e1b78e16c/core/utils_math.py#L4-L12

Reason: some utterances are about interactions with others, we need to experiment to avoid them.

This is related to: #34

• #8
• Set amount of traits to 8
• Use none for the default and filled traits for spectrum version.
• Use persona: instead of partner's persona
• Use 20 candidates instead of 6
• Fiter query/responses by miminum amount of words (in order to avoid I don't know and etc.)
• Added candidates shuffling
• Traits shuffling (to prevent from overfitting models with the specific order)

• Here we need to utilize function as a parameter for detecting speaker. Code below is a mixture of algorithm and particular API (which is relies on {, } entries):
  https://github.com/nicolay-r/chatbot_experiments/blob/55bdb9df98018c06b7c4bbf614e07a97c14a31dd/core/book/utils.py#L32-L36
• Remove legacy code (0747aad)

#14 related

• added script for assessin original amout of annotated qr pairs (non-filtered) (4349b4b)

This could be treated as a task. For features generation, there is a need to provide an amount of the related spectrums. By relying on amout, it is possible to determine the appearance frequencies in addition to the normalized values we have for features at the moment.

• saving the related data
• using norm and non-norm data for sorting
• using min-border filtering (to remove cases of rare apeared traits) [AUTO border mode]

Let's focus on the 400 speakers

Reason: we would like to exclude those who initiate dialog.

https://github.com/nicolay-r/chatbot_experiments/blob/520c115ccd4611110d82b45956434e8e1b78e16c/utils_my.py#L306-L325

#19 related.
Here is the log information:

Source of the problem:
It happens, because we first do selection of the speakers by relying on the amount of utterances related to them:
https://github.com/nicolay-r/chatbot_experiments/blob/3b83e2e8730a6d3d5b59e82671001ba135598dc9/my_s3_dataset_0_create.py#L10-L12

And then, filter some of these utterances here at dataset writing stage:
https://github.com/nicolay-r/chatbot_experiments/blob/3b83e2e8730a6d3d5b59e82671001ba135598dc9/my_s3_dataset_0_create.py#L28-L30

Reason: Already out-of-date and not supported.

Problem: at present, we use exact the same strategy with utterances selection as in the one without HLA.

https://parl.ai/docs/agent_refs/gpt3.html#

Try with book 1399.txt

https://github.com/nicolay-r/chatbot_experiments/blob/df48472db5449185d4977d67989c881a1ef0a298/core/utils_comments.py#L18

Problem: we do not clean the list of speakers.

https://github.com/nicolay-r/chatbot_experiments/blob/9e9cc64989d184fea23cad671f459347467c9e0e/my_s5_parlai_dataset_convert.py#L24-L52

This issue is related to the following case:

• iter_paragraphs_with_n_speakers -- term recognition as speakers should be function that is provided into the core as parameter