Some advice on model performance about sequence_tagging HOT 6 CLOSED

Hi Snehal,
One first remark: I noticed that you don't parse your sentence (meaning that your input is "Chicago," and not "Chicago") which makes things harder for the model in some places.

For the behavior you observe, if you are indeed learning one model on a big dataset, you may want to make sure that your dataset is balanced (meaning that sentences in the dataset are produced following the same distribution, ie in the same way). If your examples containing TECH don't have other entities, the model will un-learn what it learnt, as the loss on this example will be to predict just the TECH entity.

One solution to this problem would be to train the first model on just CoNLL. Tag your TECH dataset with it. Then, reinitialize the last layer and retrain on the whole corpus for a few epochs. Does it make sense?

from sequence_tagging.

One solution to this problem would be to train the first model on just CoNLL. Tag your TECH dataset with it.

Well, this make sense as long as all new sentences contain those other CoNLL entities, right? Which in my case looks like it doesn't! Large set of these new sentences just has one relevant entity (i.e. TECH or with CoNLL model: ORG)

reinitialize the last layer and retrain on the whole corpus for a few epochs.

I am new with NER with TensorFlow part, I would appreciate if you can point me to any code which I can modify in this repo to perform this task.

Thanks!

Edit:
Tokenized sentence gives much accurate results!

'Chicago , on Lake Michigan in Illinois , is among the largest cities in the U.S.'
[('Chicago', 'B-LOC'), (',', 'O'), ('on', 'O'), ('Lake', 'B-LOC'), ('Michigan', 'I-LOC'),
 ('in', 'O'), ('Illinois', 'B-LOC'), (',', 'O'), ('is', 'O'), ('among', 'O'),
 ('the', 'O'), ('largest', 'O'), ('cities', 'O'), ('in', 'O'), ('the', 'O'), ('U.S.', 'B-LOC')]

from sequence_tagging.

Large set of these new sentences just has one relevant entity (i.e. TECH or with CoNLL model: ORG)

You want to make sure that your training set is coherent, because the model will only learn what it sees. And if the new dataset is the same size as CoNLL, training on the combination of the two will give higher importance to TECH (appearing thus alone in half of the sentences).

I am new with NER with TensorFlow part, I would appreciate if you can point me to any code which I can modify in this repo to perform this task.

Maybe the easiest way would be build the full model at the beginning, but only train on CoNLL at first (giving it a high confidence), then, perform a few epochs to adapt to TECH (will mainly update the last layer). Does it help? (this way you don't need to modify the code, just follow the right steps).

from sequence_tagging.

Well, After training on CoNLL first and then when I try to restore session from train.py file when re-training on only "TECH" data, I am getting this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [10] rhs shape= [9]
         [[Node: save/Assign_10 = Assign[T=DT_FLOAT, _class=["loc:@proj/b"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](proj/b, save/RestoreV2_10)]]

From the basic understanding I am guessing, as the original CoNLL model has 8 labels and later I am trying to re-train with 9 labels (+1 with TECH data) this is causing the issue right?

from sequence_tagging.

Yes that's right! That's why I wrote

would be build the full model at the beginning,

meaning that you would make a prediction with all the labels! (including the TECH one even if don't see it in CoNLL)

You can also change the graph and reload only specific layers but that'll be more complicated to do in Tensorflow....

from sequence_tagging.

closing the issue.
If you come up with a better solution or have comments on it, please add a followup!

from sequence_tagging.