db_mgmt_cpe_other_dictionary.py taking a long time about cve-search HOT 20 CLOSED

Good news, the process took a while but it finished. Took around 18 hours to complete.

from cve-search.

Did you run the db_mgmt_cpe_dictionary script first? db_mgmt_cpe_other_dictionary goes over cpes in the cve info, and makes a title if none exists, so if you're running this over the entire database, this will take quite some time.

from cve-search.

Yes I did run the db_mgmt_cpe_dictionary first. This one was fast, no issues there. Let me know if there's any data I can provide. This is a test system I can play around. It's worth mentioning it's running on a VM with 4GB assigned with 4 cores assigned.

db.version()
2.6.3

from cve-search.

I'm trying to think about what the possible issue could be. I know I installed in in a vm with less RAM and cores, so that's certainly not the issue. What OS are you running? Plain Unix?

from cve-search.

I think the db_mgmt_cpe_other_dictionary should be redesigned. As this is a very huge sequential query on the dataset. I'll have a look too.

from cve-search.

It's on Ubuntu server 15.04. Used the regular packages, nothing from source. Let me know what I can do to help.

from cve-search.

tail of the mongodb log, not a log going on:

$ tail -f /var/log/mongodb/mongodb.log
2015-05-28T06:50:45.208-0700 [clientcursormon] mem (MB) res:472 virt:2351
2015-05-28T06:50:45.208-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T06:50:45.208-0700 [clientcursormon] connections:3
2015-05-28T06:55:45.268-0700 [clientcursormon] mem (MB) res:472 virt:2351
2015-05-28T06:55:45.268-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T06:55:45.268-0700 [clientcursormon] connections:3
2015-05-28T07:00:45.328-0700 [clientcursormon] mem (MB) res:472 virt:2351
2015-05-28T07:00:45.328-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T07:00:45.328-0700 [clientcursormon] connections:3
2015-05-28T07:02:04.019-0700 [conn21] end connection 127.0.0.1:40818 (2 connections now open)
2015-05-28T07:05:45.386-0700 [clientcursormon] mem (MB) res:472 virt:2351
2015-05-28T07:05:45.387-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T07:05:45.387-0700 [clientcursormon] connections:2

from cve-search.

2015-05-28T08:27:40.279-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️mozilla:firefox:32.0" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:104290 reslen:48 104ms
2015-05-28T08:29:57.932-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️apport_project:apport:2.13.3" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:107387 reslen:48 107ms
2015-05-28T08:29:58.088-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️apport_project:apport:2.14" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:132908 reslen:48 132ms
2015-05-28T08:29:58.206-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️apport_project:apport:2.14.1" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:111586 reslen:48 111ms
2015-05-28T08:29:58.622-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️apport_project:apport:2.14.6" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:113901 reslen:48 113ms
2015-05-28T08:29:59.086-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️apport_project:apport:2.16.2" } } planSummary: COLLSCAN keyUpdates:0 numYields:0 locks(micros) r:101998 reslen:48 102ms
2015-05-28T08:30:01.903-0700 [conn15] command cvedb.$cmd command: count { count: "cpe", query: { id: "cpe:2.3🅰️isc:bind:9.7.2:p3" } } planSummary: COLLSCAN keyUpdates:0 numYields:1 locks(micros) r:124491 reslen:48 131ms
2015-05-28T08:30:46.528-0700 [clientcursormon] mem (MB) res:472 virt:2351
2015-05-28T08:30:46.528-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:30:46.528-0700 [clientcursormon] connections:2
2015-05-28T08:33:19.254-0700 [conn15] build index on: cvedb.cpeother properties: { v: 1, key: { _id: 1 }, name: "id", ns: "cvedb.cpeother" }
2015-05-28T08:33:19.254-0700 [conn15] added index to empty collection
2015-05-28T08:33:20.198-0700 [conn15] insert cvedb.cpeother query: { _id: ObjectId('5567353fe45b40051f5f121b'), id: "cpe:2.3🅰️mambo:mambo_site_server:3.0.5", title: "Mambo Mambo Site Server 3.0.5" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:838564 838ms
2015-05-28T08:33:20.204-0700 [conn15] command cvedb.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:2 reslen:40 845ms
2015-05-28T08:33:21.960-0700 [conn15] insert cvedb.cpeother query: { _id: ObjectId('55673541e45b40051f5fcd9b'), id: "cpe:2.3🅰️dataparksearch:dataparksearch:4.35", title: "Dataparksearch Dataparksearch 4.35" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:179 127ms
2015-05-28T08:33:21.966-0700 [conn15] command cvedb.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:1 reslen:40 134ms
2015-05-28T08:33:23.637-0700 [conn15] insert cvedb.cpeother query: { _id: ObjectId('55673543e45b40051f60d812'), id: "cpe:2.3🅰️infrae:silva:1.6.1", title: "Infrae Silva 1.6.1" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:157305 157ms
2015-05-28T08:33:24.016-0700 [conn15] insert cvedb.cpeother query: { _id: ObjectId('55673543e45b40051f60d813'), id: "cpe:2.3🅰️infrae:silva:1.6", title: "Infrae Silva 1.6" } ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:378344 378ms
2015-05-28T08:33:24.021-0700 [conn15] command cvedb.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:164 reslen:40 544ms
2015-05-28T08:33:27.474-0700 [conn15] end connection 127.0.0.1:40811 (1 connection now open)
2015-05-28T08:33:27.474-0700 [conn14] end connection 127.0.0.1:40810 (1 connection now open)
2015-05-28T08:35:46.603-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T08:35:46.603-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:35:46.603-0700 [clientcursormon] connections:0
2015-05-28T08:40:46.659-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T08:40:46.659-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:40:46.659-0700 [clientcursormon] connections:0
2015-05-28T08:45:46.714-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T08:45:46.714-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:45:46.714-0700 [clientcursormon] connections:0
2015-05-28T08:50:46.770-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T08:50:46.770-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:50:46.770-0700 [clientcursormon] connections:0
2015-05-28T08:55:46.826-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T08:55:46.826-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T08:55:46.826-0700 [clientcursormon] connections:0
2015-05-28T09:00:46.876-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T09:00:46.876-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T09:00:46.877-0700 [clientcursormon] connections:0
2015-05-28T09:05:46.932-0700 [clientcursormon] mem (MB) res:437 virt:2351
2015-05-28T09:05:46.932-0700 [clientcursormon] mapped (incl journal view):2112
2015-05-28T09:05:46.932-0700 [clientcursormon] connections:0

from cve-search.

That's way too long to be normal... For me, it took not even 10 minutes. I tested it on Ubuntu Server 14.10, though I doubt the version would be an issue

from cve-search.

@PidgeyL Did you have the index already created?

from cve-search.

@adulau No, that script had not run yet.

from cve-search.

This is the output when the script ended:

root@db-devsrv:~/cve-search/sbin# python3 db_mgmt_cpe_other_dictionary.py
Preparing [##################################################] 70309/70309
Traceback (most recent call last):
File "db_mgmt_cpe_other_dictionary.py", line 92, in
info.update({'db': 'cpeother'}, {"$set": {'last-modified': icve['last-modified']}}, upsert=True)
TypeError: 'NoneType' object is not subscriptable

from cve-search.

@jimmyfive stop breaking it ;) I'll have a look at that. You're using the latest version, right?
When exactly did this occur? is this the first time you ran the script, on a freshly filled database, or did you do some database manipulations?

from cve-search.

lol, I'll try. :)
Yes, did a git clone

from cve-search.

@jimmyfive
Are all the issues you mentioned in this threat solved? If so, I'll close it

from cve-search.

I have the same issue running on a Ubuntu 16.04 VM. Process took more than 1 hour and still not completed.

from cve-search.

Keep in mind, we're talking about ~81300 CVE's being compared to ~117400 CPE's. It's expected to be slow, especially on a VM.
The only thing is, I am not sure we need to keep this collection. @adulau: could we not just generate the title when the info is being queried, if not found in the regular CPE database?

from cve-search.

@PidgeyL , would be nice to know from you how much time it takes in your setup (server, whatever you have).

from cve-search.

Haven't installed one in a long time, so I wouldn't be able to tell, to be honest. In most instances, I don't even use the cpe_other

from cve-search.

Stale issue message

from cve-search.