db_mgmt_cpe_dictionary.py execution takes a long time\not ending about cve-search HOT 9 CLOSED

The initial import of the CPE takes some time but 3 hours seem quite slow. I'll give a try on a test machine with the initial CPE import and let you know. Have you checked your MongoDB logs?

from cve-search.

Pasting my mongodb.log .. i can't see anything suspicious

roy@ubuntu:/var/log/mongodb$ more mongodb.log
Mon Aug 24 08:42:43.249 [initandlisten] MongoDB starting : pid=9729 port=27017 dbpath=/var/lib/mongodb 64-bit host=ubuntu
Mon Aug 24 08:42:43.250 [initandlisten] db version v2.4.9
Mon Aug 24 08:42:43.250 [initandlisten] git version: nogitversion
Mon Aug 24 08:42:43.250 [initandlisten] build info: Linux orlo 3.2.0-58-generic #88-Ubuntu SMP Tue Dec 3 17:37:58 UTC 2013 x86_64 BOOST_LIB_VERSION=1_54
Mon Aug 24 08:42:43.250 [initandlisten] allocator: tcmalloc
Mon Aug 24 08:42:43.250 [initandlisten] options: { bind_ip: "127.0.0.1", config: "/etc/mongodb.conf", dbpath: "/var/lib/mongodb", journal: "true", logappend: "true", logpath: "/var/log/mongodb/mongodb.log
" }
Mon Aug 24 08:42:43.254 [initandlisten] journal dir=/var/lib/mongodb/journal
Mon Aug 24 08:42:43.254 [initandlisten] recover : no journal files present, no recovery needed
Mon Aug 24 08:42:43.340 [FileAllocator] allocating new datafile /var/lib/mongodb/local.ns, filling with zeroes...
Mon Aug 24 08:42:43.340 [FileAllocator] creating directory /var/lib/mongodb/_tmp
Mon Aug 24 08:42:43.342 [FileAllocator] done allocating datafile /var/lib/mongodb/local.ns, size: 16MB, took 0 secs
Mon Aug 24 08:42:43.342 [FileAllocator] allocating new datafile /var/lib/mongodb/local.0, filling with zeroes...
Mon Aug 24 08:42:43.342 [FileAllocator] done allocating datafile /var/lib/mongodb/local.0, size: 64MB, took 0 secs
Mon Aug 24 08:42:43.389 [websvr] admin web console waiting for connections on port 28017
Mon Aug 24 08:42:43.389 [initandlisten] waiting for connections on port 27017
Mon Aug 24 08:43:50.524 [initandlisten] connection accepted from 127.0.0.1:48012 #1 (1 connection now open)
Mon Aug 24 08:43:50.530 [initandlisten] connection accepted from 127.0.0.1:48013 #2 (2 connections now open)
Mon Aug 24 08:44:14.430 [FileAllocator] allocating new datafile /var/lib/mongodb/cvedb.ns, filling with zeroes...
Mon Aug 24 08:44:14.431 [FileAllocator] done allocating datafile /var/lib/mongodb/cvedb.ns, size: 16MB, took 0 secs
Mon Aug 24 08:44:14.431 [FileAllocator] allocating new datafile /var/lib/mongodb/cvedb.0, filling with zeroes...
Mon Aug 24 08:44:14.432 [FileAllocator] done allocating datafile /var/lib/mongodb/cvedb.0, size: 64MB, took 0 secs
Mon Aug 24 08:44:14.432 [FileAllocator] allocating new datafile /var/lib/mongodb/cvedb.1, filling with zeroes...
Mon Aug 24 08:44:14.433 [FileAllocator] done allocating datafile /var/lib/mongodb/cvedb.1, size: 128MB, took 0 secs
Mon Aug 24 08:44:14.433 [conn2] build index cvedb.cves { _id: 1 }
Mon Aug 24 08:44:14.434 [conn2] build index done. scanned 0 total records. 0 secs
Mon Aug 24 08:44:14.766 [conn2] insert cvedb.cves ninserted:6720 keyUpdates:0 locks(micros) w:336021 336ms
Mon Aug 24 08:44:20.404 [conn2] insert cvedb.cves ninserted:4685 keyUpdates:0 locks(micros) w:246912 246ms
Mon Aug 24 08:44:24.200 [conn2] insert cvedb.cves ninserted:7047 keyUpdates:0 locks(micros) w:303691 303ms
Mon Aug 24 08:44:27.367 [FileAllocator] allocating new datafile /var/lib/mongodb/cvedb.2, filling with zeroes...
Mon Aug 24 08:44:27.368 [FileAllocator] done allocating datafile /var/lib/mongodb/cvedb.2, size: 256MB, took 0 secs
Mon Aug 24 08:44:52.368 [conn2] insert cvedb.cves ninserted:7033 keyUpdates:0 locks(micros) w:375625 375ms
Mon Aug 24 08:44:56.125 [conn2] insert cvedb.cves ninserted:4886 keyUpdates:0 locks(micros) w:300319 300ms
Mon Aug 24 08:45:02.006 [conn2] insert cvedb.cves ninserted:4949 keyUpdates:0 locks(micros) w:683031 683ms
Mon Aug 24 08:45:14.340 [FileAllocator] allocating new datafile /var/lib/mongodb/cvedb.3, filling with zeroes...
Mon Aug 24 08:45:14.342 [FileAllocator] done allocating datafile /var/lib/mongodb/cvedb.3, size: 512MB, took 0.001 secs
Mon Aug 24 08:45:14.358 [conn2] insert cvedb.cves ninserted:2621 keyUpdates:0 locks(micros) w:902044 902ms
Mon Aug 24 08:45:19.850 [conn2] insert cvedb.cves ninserted:5191 keyUpdates:0 locks(micros) w:115126 115ms
Mon Aug 24 08:47:32.562 [conn2] end connection 127.0.0.1:48013 (1 connection now open)
Mon Aug 24 08:47:32.562 [conn1] end connection 127.0.0.1:48012 (1 connection now open)
Mon Aug 24 08:48:25.889 [initandlisten] connection accepted from 127.0.0.1:48028 #3 (1 connection now open)
Mon Aug 24 08:48:26.316 [initandlisten] connection accepted from 127.0.0.1:48030 #4 (2 connections now open)
Mon Aug 24 08:48:53.328 [conn4] build index cvedb.cpe { _id: 1 }
Mon Aug 24 08:48:53.329 [conn4] build index done. scanned 0 total records. 0 secs
Mon Aug 24 09:00:11.418 [conn4] update cvedb.cpe query: { id: "cpe:2.3:a:nlnetlabs:ldns:1.6.10" } update: { $set: { cpe_2_2: "cpe:/a:nlnetlabs:ldns:1.6.10", title: "nlnetlabs ldns 1.6.10" } } nscanned:718
41 nupdated:1 fastmodinsert:1 keyUpdates:0 numYields: 1 locks(micros) w:33290 123ms
Mon Aug 24 09:07:02.570 [conn4] update cvedb.cpe query: { id: "cpe:2.3:a:steindom:contact_save:6.x-1.2:beta2" } update: { $set: { cpe_2_2: "cpe:/a:steindom:contact_save:6.x-1.2:beta2", title: "steindom Co
ntact Save (contact_save) module for Drupal 6.x-1.2 beta2" } } nscanned:91753 nupdated:1 fastmodinsert:1 keyUpdates:0 locks(micros) w:101209 101ms
Mon Aug 24 09:07:39.033 [conn4] end connection 127.0.0.1:48030 (1 connection now open)
Mon Aug 24 09:07:39.520 [conn3] end connection 127.0.0.1:48028 (0 connections now open)
Mon Aug 24 09:36:47.554 [initandlisten] connection accepted from 127.0.0.1:48093 #5 (1 connection now open)
Mon Aug 24 09:36:48.343 [initandlisten] connection accepted from 127.0.0.1:48095 #6 (2 connections now open)
Mon Aug 24 09:37:31.944 [conn6] update cvedb.cpe query: { id: "cpe:2.3:a:adiscon:loganalyzer:3.4.0" } update: { $set: { cpe_2_2: "cpe:/a:adiscon:loganalyzer:3.4.0", title: "Adiscon LogAnalyzer 3.4.0" } }
nscanned:93295 nupdated:1 keyUpdates:0 numYields: 1 locks(micros) w:116503 107ms
roy@ubuntu:/var/log/mongodb$

from cve-search.

initially, it looks like this should be working. Are you, by any chance, using a slow medium to store your data (like a flashdrive)? I used to do that, so I know it takes a very long time in that case.

from cve-search.

@PidgeyL i'm using a VM on a W540 laptop with SSD, 32bit (4 gb of which are assigned to the VM), as well as 4 CPUs. the OS is ubuntu 14 LTS.
roy@ubuntu:~/Desktop/cve-search-pidgin/cve-search/sbin$ uname -a && cat /etc/issue

Linux ubuntu 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 14.04.3 LTS \n \l

i'll give it a go again and come back to you with results.

from cve-search.

ok, this issue relates to the first source in the update scripts. looks like nist is throttling traffic, not sure if it was because i reran the script alot of times, or some bandwith limitations from my office. i noticed that it would't run my house. as soon as i connected to a vpn it started working..

from cve-search.

good to know your issue is solved :)

from cve-search.

this looks similar to my issue. However, I could not have it resolved on either my home or in the office. Could any of you guys provide more details regarding how can I modify the update scripts in order to have the data populated? thanks

from cve-search.

If the issue is the network connection, you can download the files manually and make the config file point to those files

from cve-search.

Hello, I think i have the same issue, can you tell me exactly where i found the files to download them manually?

thanks

from cve-search.