Abstract definition of browser tarballs about rules_webtesting HOT 2 CLOSED

But internally when we decide to update "//testing/web/browsers:chrome-linux" to a new browser version, we can identify all users who the change breaks and notify them of the breakage (and potentially even fix the breakage). In the open source case that is much more difficult, and when I push out new versions of rules_web, I don't want to see people delaying adopting them because it happens to switch a browser version to a version that breaks their tests (and similarly when a new browser version is pushed out, I don't want users to delay adopting it because I haven't updated rules_web, and I don't want to make new releases of rules_web just because a new browser version is available).

And even if I did include the version numbers in the browser, which versions should I include? When can I delete a reference to an old version from the repository (or do I have to keep every version forever)? How do I deal with browsers with multiple channels with different release schedules (e.g. Chrome has at least stable, beta, and dev channels, and I can see valid arguments for testing against any or all of these)?

In Google, my team has made decisions about handling all of these issues, but those decisions were possible because we can readily identify every user of every browser we have defined. And even with that, we get flack for the decisions we make (everybody always seem to want more browser versions to test against, and always complain when by testing against more browser version they see increase flakiness) and sometimes make mistakes (i.e. deleting a browser version while it still has users, or upgrading a browser version to one that introduces new problems for users). And we essentially use 1 full-time head count (on rotation through our team) to manage browser updates.

from rules_webtesting.

You're absolutely right. We don't have the global choo-choo for testing on GitHub. So I would honestly just keep the old versions around forever. If those old browsers aren't being used, the only cost should be parsing the http_file and adding it to a hash map.

In order to make this happen, we would ideally want a distinct package for each version of each browser. We would probably want the version numbers to be the same as caniuse.com which will be intuitive to users.

So maybe what I would say is:

web_test_suite(
    name = "browser_test",
    browsers = [
        "@io_bazel_rules_web//browsers/chromium/v54",
        "@io_bazel_rules_web//browsers/not_hermetically_sealed/firefox/v49",
    ],
    local = 1,
    test = ":browser_test_wrapped",
)

from rules_webtesting.