Gemfile parsing for Bundler (Ruby) doesn't correctly take into account platforms (ruby, java etc.) about ort HOT 9 OPEN

Not sure if it helps, but I toyed around with it a bit more and the minimal Gemfile with which I can trigger the problem seems to be this:

source 'https://rubygems.org'

gem 'rails'
gem 'rails-i18n'

rails depends on railties, which depends on irb, which depends on rdoc, which depends on psych, which has a dependency on jar-dependencies on the Java platform, but not others.

And rails-i18n has a dependency on rails and then consequently the same transitive dependencies.

When I remove one or the other, the problem doesn't appear. But when I have both in my Gemfile, the problem is there.

The same is true when I replace either of them with vite_rails (= another package that depends on rails).

But the problem doesn't appear when I put in 2 packages that both depend on rdoc, but otherwise don't share transitive dependencies; e.g. this is fine:

source 'https://rubygems.org'

gem 'irb'
gem 'minitest'

from ort.

Thanks for the sophisticated report. In essence, this is a specific case of #4013.

from ort.

I guess part of the problem is this line:

from ort.

I haven't traced the whole call chain and I'm by no means a Kotlin expert, but based on the stack trace, the problem is in this function:

It seems that independent of that Ruby script that uses Bundler's own facilities to resolve dependencies, the RubyGems API is still queried directly, using ORT's HTTP client (at least that's how I'm interpreting okHttpClient.downloadText(url)).

This is also evidenced by the fact that ORT seems to be trying to resolve a dependency for the java version of the psych gem, even though Bundler is set to force the generic (= non-java) version for resolving.

If ORT were to solely rely on Bundler's mechanisms instead of querying the API manually, I think it might generally be fine, but you might want to either remove the Bundler.settings.set_global(:force_ruby_platform, true) (which could cause wrong results for basically any package with native extensions) or at least offer some kind of configuration option to allow to opt out of this. Because Bundler actually behaves correctly by default, IMO – see https://github.com/ruby/ruby/blob/e49d68bf2732cf467a1532283de4a1716e238677/lib/bundler/definition.rb#L634.

from ort.

Further digging: According to the source code of rubygems.org, it is expected behavior that when not explicitly specifying platform, it picks whatever was last created.

For psych, this seems to be the Java version:

curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml?platform=ruby'
...
created_at: '2023-12-19T02:06:51.830Z'
...

curl 'https://rubygems.org/api/v2/rubygems/psych/versions/5.1.2.yaml'
...
created_at: '2023-12-19T02:07:03.237Z'
...

Whereas e.g. for nokogiri (another multi-platform gem), it's Ruby:

curl 'https://rubygems.org/api/v2/rubygems/nokogiri/versions/1.16.5.yaml'
...
created_at: '2024-05-13T14:01:29.505Z'
...

curl 'https://rubygems.org/api/v2/rubygems/nokogiri/versions/1.16.5.yaml?platform=java'
...
created_at: '2024-05-13T14:01:04.168Z'
...

This still doesn't explain why sometimes it seems to be fine and sometimes it isn't (see my previous comment), but at least indicates that if ORT wants to have deterministic behavior, the platform needs to always be explicitly specified. (And which platform it is can be derived from Gemfile.lock only.)

from ort.

Any news on this one, @sschuberth ? Can I help with further info?

Also, if the intention was to enforce ruby as a platform and it's just not working for that one part of the codebase, would it make sense to adapt this accordingly until a more elaborate fix (which takes into account all declared platforms) is in place?

from ort.

Any news on this one, @sschuberth ? Can I help with further info?

I believe your info is more than sufficient, thank you @clemens. It's just that to be best of my knowledge no one from the community is planning to work on this (for free) in the near future. If you're interested in a commercial ORT support subscription that could accelerate this (also see #8381), feel free to contact any of the mentioned parties directly.

from ort.

That's understandable. :)

Would you accept an MR that appends ?platform=ruby to the problematic URL so that the problem is at least consistent and in line what you apparently intended with

from ort.

Would you accept an MR that appends ?platform=ruby to the problematic URL

Sounds reasonable. But for me it's the easiest to tell from the diff of a MR / PR, as that would need to include any necessary changes to the expected test results.

from ort.