Comments (9)
This is a pulpcore issue. We can move it there.
from pulpcore.
The query that is being run when this happens is:
SELECT COUNT(*) FROM (SELECT DISTINCT "core_content"."pulp_id" AS "col1", "core_content"."pulp_created" AS "col2", "core_content"."pulp_last_updated" AS "col3", "core_content"."pulp_type" AS "col4", "core_content"."upstream_id" AS "col5", "core_content"."timestamp_of_interest" AS "col6", "core_content"."pulp_domain_id" AS "col7", "rpm_package"."content_ptr_id" AS "col8", "rpm_package"."name" AS "col9", "rpm_package"."epoch" AS "col10", "rpm_package"."version" AS "col11", "rpm_package"."release"
from pulpcore.
I believe this is happening because our REST API always returns how many total items there in the table. Example response.
{"count":104645,"next":"http://pulp.stage.devshift.net/api/pulp/default/api/v3/repositories/rpm/rpm/?fields=name&limit=25&offset=25","previous":null,"results":[{"name":"praiskup/ping+custom+foo/fedora-39-aarch64"},{"name":"praiskup/ping+custom+dir/epel-6-x86_64"},{...
from pulpcore.
@dkliban Do we know that this is actually tied somehow to the RPM plugin, or is it something that can happen to any plugin with enough content?
Do we know why it's running a distinct on all of these different fields? I would think that in theory it would only need to count the number of packages associated with the domain.
from pulpcore.
There's nothing from pulp_rpm in that traceback, so I feel like whatever is happening here, it's a more-global problem.
from pulpcore.
@dkliban
It might be installation specific, apparently workers times-out if query takes longer than 90secs?
Here's the output from the installation with 4.2Mlns RPMs
$ time pulp rpm content list --limit 1
Not all 4288330 entries were shown.
[
{
"pulp_href": "/pulp/api/v3/content/rpm/packages/018fe33f-da80-798a-89f0-974857ec38c2/",
"pulp_created": "2024-06-04T12:36:21.913133Z",
"pulp_last_updated": "2024-06-04T12:36:21.913145Z",
"md5": null,
"sha1": null,
"sha224": null,
"sha256": null,
"sha384": null,
"sha512": null,
"artifact": null,
"name": "rubygem-ziptz",
"epoch": "0",
"version": "5.1.1",
"release": "1.fc41",
"arch": "src",
"pkgId": "854b9ccb2e7af74564218ccf93ba0c55128350a1d30dd036e56b2aa3e7777333",
"checksum_type": "sha256",
"summary": "TimeZone info for any 5-digit US zip code",
"description": "Get timezone info for all 5-digit US zip codes.",
"conflicts": [],
"obsoletes": [],
"suggests": [],
"enhances": [],
"recommends": [],
"supplements": [],
"location_base": "",
"location_href": "07397794-rubygem-ziptz/rubygem-ziptz-5.1.1-1.fc41.src.rpm",
"rpm_buildhost": "copr-hv-x86-64-01-prod-06811773-20240502-040521",
"rpm_group": "Unspecified",
"rpm_license": "MIT",
"rpm_packager": "",
"rpm_sourcerpm": "",
"rpm_vendor": "Fedora Copr - group @rubygems",
"rpm_header_start": 5112,
"rpm_header_end": 8745,
"is_modular": false,
"size_archive": 551684,
"size_installed": 551297,
"size_package": 441888,
"time_build": 1714623017,
"time_file": 1714623030
}
]
real 0m16.087s
user 0m0.389s
sys 0m0.050s
from pulpcore.
"md5": null,
"sha1": null,
"sha224": null,
"sha256": null,
"sha384": null,
"sha512": null,
"artifact": null,
uhhh is that a bug
edit: same as pulp/pulp_rpm#3145
it's not a bug exactly, the API doesn't show the "remote artifact" checksums, but it is weird as a user to see that
from pulpcore.
@ipanova it might depend on the makeup of the RPMs. COPR RPMs would have very few dependencies set compared to RHEL repos.
from pulpcore.
"md5": null, "sha1": null, "sha224": null, "sha256": null, "sha384": null, "sha512": null, "artifact": null,
uhhh is that a bug
edit: same as pulp/pulp_rpm#3145
it's not a bug exactly, the API doesn't show the "remote artifact" checksums, but it is weird as a user to see that
yes, that's on on-demand sync, seems like pulp/pulp_rpm#3145 indeed
from pulpcore.
Related Issues (20)
- Add an configuration option to cap task group workers usage
- urlize_quoted_hrefs() tag doesn't make links any more
- Provide a general compatibility matrix for plugins HOT 2
- Fix aiofiles UB in 3.22 to <23.3.0 instead of <= HOT 1
- Breaking changes in 3.49.14 Ruby bindings HOT 14
- Provide general feature and workflow matrix for plugins HOT 3
- Permission error when trying to use `has_repo_or_repo_ver_param_model_or_obj_perms` function
- default domain does not respect REDIRECT_TO_OBJECT_STORAGE HOT 1
- Upload by url doesn't work for certain plugins
- Error when reclaiming space from repository with over 65k packages due to parameter limit
- urlize_quoted_hrefs() still broken on v3 API browser
- RBAC permissions incorrectly assigned in tasks that create objects
- pulp_file fails to sync file repos literally containing URL encoding strings like %3a in a file name HOT 4
- Expose `no_content_change_since` field on the Distribution serializer HOT 1
- pulp-workers should log which domain a task was executed in
- API Documentation does not explain the workings of the q parameter
- [3.49.15] drf browser showing html tags for url-like fields
- support orphan cleanup within a domain HOT 3
- How to make file repositories private HOT 3
- Wrong package download after on-demand sync
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pulpcore.