Git Product home page Git Product logo

Comments (9)

dkliban avatar dkliban commented on August 20, 2024 1

This is a pulpcore issue. We can move it there.

from pulpcore.

dkliban avatar dkliban commented on August 20, 2024

The query that is being run when this happens is:

SELECT COUNT(*) FROM (SELECT DISTINCT "core_content"."pulp_id" AS "col1", "core_content"."pulp_created" AS "col2", "core_content"."pulp_last_updated" AS "col3", "core_content"."pulp_type" AS "col4", "core_content"."upstream_id" AS "col5", "core_content"."timestamp_of_interest" AS "col6", "core_content"."pulp_domain_id" AS "col7", "rpm_package"."content_ptr_id" AS "col8", "rpm_package"."name" AS "col9", "rpm_package"."epoch" AS "col10", "rpm_package"."version" AS "col11", "rpm_package"."release" 

from pulpcore.

dkliban avatar dkliban commented on August 20, 2024

I believe this is happening because our REST API always returns how many total items there in the table. Example response.

{"count":104645,"next":"http://pulp.stage.devshift.net/api/pulp/default/api/v3/repositories/rpm/rpm/?fields=name&limit=25&offset=25","previous":null,"results":[{"name":"praiskup/ping+custom+foo/fedora-39-aarch64"},{"name":"praiskup/ping+custom+dir/epel-6-x86_64"},{...

from pulpcore.

dralley avatar dralley commented on August 20, 2024

@dkliban Do we know that this is actually tied somehow to the RPM plugin, or is it something that can happen to any plugin with enough content?

Do we know why it's running a distinct on all of these different fields? I would think that in theory it would only need to count the number of packages associated with the domain.

from pulpcore.

ggainey avatar ggainey commented on August 20, 2024

There's nothing from pulp_rpm in that traceback, so I feel like whatever is happening here, it's a more-global problem.

from pulpcore.

ipanova avatar ipanova commented on August 20, 2024

@dkliban
It might be installation specific, apparently workers times-out if query takes longer than 90secs?

Here's the output from the installation with 4.2Mlns RPMs

$ time pulp rpm content list --limit 1                                                                                                                                                              
Not all 4288330 entries were shown.
[                          
  {                     
    "pulp_href": "/pulp/api/v3/content/rpm/packages/018fe33f-da80-798a-89f0-974857ec38c2/",                            
    "pulp_created": "2024-06-04T12:36:21.913133Z",
    "pulp_last_updated": "2024-06-04T12:36:21.913145Z",
    "md5": null,             
    "sha1": null,          
    "sha224": null,
    "sha256": null,
    "sha384": null,                                        
    "sha512": null,
    "artifact": null,
    "name": "rubygem-ziptz",
    "epoch": "0",                           
    "version": "5.1.1",
    "release": "1.fc41",
    "arch": "src",
    "pkgId": "854b9ccb2e7af74564218ccf93ba0c55128350a1d30dd036e56b2aa3e7777333",
    "checksum_type": "sha256",
    "summary": "TimeZone info for any 5-digit US zip code", 
    "description": "Get timezone info for all 5-digit US zip codes.",
    "conflicts": [],
    "obsoletes": [],
    "suggests": [],
    "enhances": [],
    "recommends": [],
    "supplements": [],
    "location_base": "",
    "location_href": "07397794-rubygem-ziptz/rubygem-ziptz-5.1.1-1.fc41.src.rpm",
    "rpm_buildhost": "copr-hv-x86-64-01-prod-06811773-20240502-040521",
    "rpm_group": "Unspecified",
    "rpm_license": "MIT",
    "rpm_packager": "",
    "rpm_sourcerpm": "",
    "rpm_vendor": "Fedora Copr - group @rubygems",
    "rpm_header_start": 5112,
    "rpm_header_end": 8745,
    "is_modular": false,
    "size_archive": 551684,
    "size_installed": 551297,
    "size_package": 441888,
    "time_build": 1714623017,
    "time_file": 1714623030
  }
]

real    0m16.087s
user    0m0.389s
sys     0m0.050s

from pulpcore.

dralley avatar dralley commented on August 20, 2024
    "md5": null,             
    "sha1": null,          
    "sha224": null,
    "sha256": null,
    "sha384": null,                                        
    "sha512": null,
    "artifact": null,

uhhh is that a bug

edit: same as pulp/pulp_rpm#3145

it's not a bug exactly, the API doesn't show the "remote artifact" checksums, but it is weird as a user to see that

from pulpcore.

dralley avatar dralley commented on August 20, 2024

@ipanova it might depend on the makeup of the RPMs. COPR RPMs would have very few dependencies set compared to RHEL repos.

from pulpcore.

ipanova avatar ipanova commented on August 20, 2024
    "md5": null,             
    "sha1": null,          
    "sha224": null,
    "sha256": null,
    "sha384": null,                                        
    "sha512": null,
    "artifact": null,

uhhh is that a bug

edit: same as pulp/pulp_rpm#3145

it's not a bug exactly, the API doesn't show the "remote artifact" checksums, but it is weird as a user to see that

yes, that's on on-demand sync, seems like pulp/pulp_rpm#3145 indeed

from pulpcore.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.