Where is the benchmark about 90 times faster than HF transformers? about tf-transformers HOT 11 CLOSED

In addition, relative difference can't be expressed as times.
relative difference means the related numerical difference of two numbers.

For example, relative difference of 1 and 2 is 66.7%
is 2kg 66 times heavier than 1kg?
is 2cm 66 times longer than 1cm?
it's not true.

2 is 2 times larger than 1.
66 is 66 times larger than 1.
1 is 66.7% different with 2.
2 is 66.7% different with 1.

why did you express the % as times?

from tf-transformers.

I admire you for developing such a wonderful library. However, skewing the benchmarks is a problem, and it makes users untrustworthy of this library. I recommend that you correct any incorrect expressions in the documentation.

from tf-transformers.

Hi @hyunwoongko . Thanks for opening the first issue.
You can find it here .

I tested it on V100 GPU machine.

Code to test: https://github.com/legacyai/tf-transformers/tree/main/benchmarks/gpt2
Website docs : https://legacyai.github.io/tf-transformers/build/html/benchmarks/gpt2.html

Please ignore (tf_text) part .

from tf-transformers.

Yeah. It's much faster than HF tensorflow implementation.
But 31 minutes is not 90 times faster than 83 minutes. no?
Is there anything I misunderstood?

from tf-transformers.

I have calculated relative difference . From 83 minutes to 31 minutes.

Online calculator I used.
https://www.easycalculation.com/algebra/relative-percentage-difference-calculator.php
To be accurate it is 91.228x faster.

But, trust me, in many cases I have even seen 120x times speedup.
But for that other factors comes into picture, variable batch decoding, eos token, sequence length etc.

I have. tutorial ( https://legacyai.github.io/tf-transformers/build/html/tutorials/7_gpt2_question_answering_squad.html ) here, you can experience similar speedup.

from tf-transformers.

nobody thinks 31 minutes is 90 times faster than 83 minutes. it's 2.67x faster.

from tf-transformers.

Agreed . I always calculated relative difference to measure, and apologies for wrong usage of word times as such. I always had this in my mind and updated the docs with new information. From 90 times to 90 % improvement.

from tf-transformers.

I admire you for developing such a wonderful library. However, skewing the benchmarks is a problem, and it makes users untrustworthy of this library. I recommend that you correct any incorrect expressions in the documentation.

I don't think, it was supposed to make users untrustworthy or skewed benchmarks. The numbers provided were correct, just that I used times in one part of the README incorrectly. If thats the case, I wouldn't have provided all the scripts and code to reproduce benchmark. But, yes It was mistake from my end for the wrong usage of wordtimes. :)

from tf-transformers.

If I am not wrong relative difference is a valid measure to calculate the speedup right. Yes, it can't be expressed as times, my bad. But I believe the benchmarking is very much valid.

from tf-transformers.

Just want to second what @hyunwoongko said, the difference from 83min -> 31min should be expressed as either:

• (31/83) = 0.37 -> decoding takes 37% of the time as it did before
• or (83/31) = 2.67 -> decoding is performed 2.67x faster than before

The relative difference = |a - b| / mean(a, b) does not make any sense to me as a measure for comparing minutes or throughputs. Unlike the two meaures above, relative difference does not give me a simple way to predict how much faster my own workload would get if I used your library. I would rather not have to solve b = a * (2 - relative_difference) / (2 + relative_difference) in my head.

from tf-transformers.

@aveni - This is tutorial from huggingface https://huggingface.co/blog/tf-serving .

If you see they expressed the speedup in terms of relative difference. I checked internet for how to measure speedup and relative difference seems to be a valid one. But I could be wrong too, I followed it because it was used in the above blog officially and it gives as sense of percentage improvement also.

from tf-transformers.