Older version (2020.6.4) of PSPDFKIT served by the benchmark? about pspdfkit-webassembly-benchmark HOT 6 CLOSED

Note: I have also tried to run the benchmark locally but even with older versions than the most recent one it fails at the 3rd step (Export ) with:

PSPDFKitError: Annotations: `text` is not an object with properties `format` and `value`

from pspdfkit-webassembly-benchmark.

Hello Matthias. It's great to see that the benchmark is proving useful for validating performance improvements in V8.

What you're observing here is surprising to me as we intentionally left a the benchmark on 2020.6.4 to have results be as comparable as possible between test runs. We mistakenly updated the version once during a release in 2021, but that change was quickly reverted and according to our git history there were no other explicit updates done from our side. I'll take a look internally to see if I can find out more.

That being said, while we do believe that constantly updating our WASM version would result in a less useful benchmark overall, I do acknowledge that the used version is now very outdated. We're considering doing a one time update and releasing what would essentially be a v2 of the benchmark with our latest release and WASM binary. With that we can also check out and see what's going on with that error you encountered.

Would this work for you and if so, what kind of timeline are you operating under for your performance improvements?

from pspdfkit-webassembly-benchmark.

I'll take a look internally to see if I can find out more.

Thanks, I'd be certainly interested in case you find something out.
We experimented with Inlining about half a year ago and saw similar improvements as I did with the supposedly more recent version.

We're considering doing a one time update and releasing what would essentially be a v2 of the benchmark with our latest release and WASM binary. [...]
Would this work for you and if so, what kind of timeline are you operating under for your performance improvements?

That would be amazing.
For my current project I plan on finching (meaning releasing for parts of the user base) the Inlining feature as part of Chrome 123 which will hit production in about 4 weeks.
The main disadvantage is that we will mainly see the (background) compilation time metric (regressing). We might have less data points on the performance gains which is why there we have to rely at least partially on benchmarks where we have seen very different results (though no regressions on peak performance).

If I could fully run the benchmark locally, that would already be great.
Having an official PSPDFKit wasm benchmark v2 would still be very nice as the old version is great for long term comparisons but arguably the performance of a 4 year old application is less relevant than the performance of a current one.

from pspdfkit-webassembly-benchmark.

We went ahead and updated the benchmark. We already pushed the changes to this repository and https://pspdfkit.com/webassembly-benchmark/ should reflect the changes soon as well. You'll see a new footer that will indicate that PSPDFKit for Web 2024.1.0 is being used.

I hope this helps. I'd love it if you could tell us what kind of improvements you're seeing with this new deployment.

from pspdfkit-webassembly-benchmark.

Thanks a lot! (I realized, I didn't reply here, yet.)
It's nice to have one more data point with this.
We currently test the inlining for wasm with linear memory on 50% chrome beta but I don't have any results yet.
Locally, the compile time regression for inlining with the new benchmark is lower but the benefits are lower as well.

For the old module we see a 5-10% performance improvement depending on architecture.
On the extreme side we have seen a 40% background compile time regression on Mac m1 mini, it mostly stays around 20%.
On M1 mini we see the following runtime changes:

• Searching: -33%
• Exporting: -32%
• Annotations: -25%
• Rendering: -4%
• Total: -8%

So there are huge improvements on the "smaller" parts of the benchmarks and only a small improvement on rendering.
I'll push for integrating the new benchmark into our benchmark suite as well, so I should get numbers on every platform for it as well, as said, locally I have seen smaller runtime improvements but also smaller compile time regressions (probably indicating that the newer version has better inlining decisions already happening during build time of the PSPDFKIT wasm module than the old module).

from pspdfkit-webassembly-benchmark.

Thank you for the update. I'm going to close the ticket, but please don't hesitate to chime back in if you need anything else of if you might have other results to share in the future.

from pspdfkit-webassembly-benchmark.