cve-search / cve-search Goto Github PK
View Code? Open in Web Editor NEWcve-search - a tool to perform local searches for known vulnerabilities
Home Page: https://www.cve-search.org/
License: GNU Affero General Public License v3.0
cve-search - a tool to perform local searches for known vulnerabilities
Home Page: https://www.cve-search.org/
License: GNU Affero General Public License v3.0
If the length of the collection is zero then there is ZeroDivisionError.
I faced this issue in db_mgmt_cpe_other_dictionary.py when updating
the database.
On line 67 it is checked that list(collections) is not empty.
It seems, however, that the length may be still be 0.
I do not know whether the proper correction is to add testing of
zero length. Anyway, it removed the ZeroDivisionError.
Br,
Raino
I found a weird issue with the XMPP client. When I use a command that accesses the API, I get no return. I put some prints in there to see where it stops, and I couldn't quite find it. The api searcher returns the CVE (when i use get) all the way up to def message
, but I get no return. I'm using ejabberd. Can you reproduce this issue? @adulau
Should we put the query interfaces (XMPP bot, IRC bot, API, searcy.py, dump_last.py, ...) in a different folder or not? Would it be a good idea to split database scripts (back end) from the interfaces (front end)?
There are other vulnerability database that we could import into cve-search but those vulnerabilities have often no CVE id assigned (temporary or perpetually) . We should find a clean way to import those and reference back the vulnerabilities (to CPE as an example) even if there is no CVE id.
The idea is to build a new collection where the vulnerabilities without CVE id are imported but referenced with their CPE.
It's an open question. cc/ @PidgeyL
This would improve the speed of some actions like adding/updating and/or removing CPEs
Hi,
I have an issu while running 'python3 index.py' , and it's th same with minimal-web.py
Cmdline Paste :
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 111, in
from collections.abc import Generator as GeneratorType # py35+
ImportError: cannot import name 'Generator'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "index.py", line 19, in
from tornado.wsgi import WSGIContainer
File "/usr/local/lib/python3.4/dist-packages/tornado/wsgi.py", line 42, in
from tornado import web
File "/usr/local/lib/python3.4/dist-packages/tornado/web.py", line 84, in
from tornado import gen
File "/usr/local/lib/python3.4/dist-packages/tornado/gen.py", line 113, in
from backports_abc import Generator as GeneratorType
ImportError: No module named 'backports_abc'
I have installed tornado, but message still appear.
I also have an error when i tried to install lxml with "pip3 install -r requirement.txt"
Cmdline Paste :
Command /usr/bin/python3 -c "import setuptools, tokenize;file='/tmp/pip-build-42c055cm/lxml/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-b5_m1i3k-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-build-42c055cm/lxml
Storing debug log for failure in /root/.pip/pip.log
i run on Debian 8
Could you help me ?
Thanks
I would like to make an IRC bot for CVE-Search. Do I make this within CVE-Search, like the XMPP client, or do I make this a separate project?
formalizing the CPE format would allow for whitelisting of target software/hardware or other tags, which can be very useful. Example:
(1) WFN:
wfn:[
part="o",
vendor="microsoft",
product="windows_vista",
version="6\.0",
update="sp1",
edition=NA,
language=NA,
sw_edition="home_premium",
target_sw=NA,
target_hw="x64",
other=NA
]
(2) WFN bound to a URI:
cpe:/o:microsoft:windows_vista:6.0:sp1:~-~home_premium~-~x64~-
(3) WFN bound to a formatted string:
cpe:2.3:o:microsoft:windows_vista:6.0:sp1:-:-:home_premium:-:x64:-
(4) Current Format (WFN URI-like notation):
cpe:/o:microsoft:windows_vista:6.0:sp1:~~home_premium~~x64~
In our case, the third option would be better. This way, we can more formal regexes.
The current setup still allows for regex searching (for example by target software or hardware), but is not the nicest. Worth changing?
@adulau @wimremes
It would be a good idea to make CVE-Search modular and customizable with plug-ins.
Examples of plugins would be:
I am concidering importing the partial cvss score (impact and exploit) into the database. It is not in our NVD Source list, but it is in vFeed.
I would like a second opinion on the implementation of this:
Hi all,
I've just installed cve-search. It works like a charm, I can make queries by script...
However I can't enable Fulltext search:
Fulltext search not enabled
The current settings in the database do not allow fulltext search.
If you feel like this should be enabled, please contact your administrator.
However, search text is enable on mongodb by default since version 3.
I tried to apply this command , but the result is the same:
db.adminCommand({"setParameter": 1, "textSearchEnabled":true})
return this:
{ "was" : true, "ok" : 1 }
Could you please, help me?
Regards,
Hello
It would be possible to add fulltext visualization (search_fulltext.py -g -s) to products and vendors?
I wish make build a list of the most common products and venders, no keywords used in CVE.
They also do not know if it's possible to use -g -s but adding a filter or external command.
Example:
python3 bin/search_fulltext.py -g -s | jq 'select ((.Modified >= "2015-01-01" and .Modified <="2015-12-31")) | >cve.json
This I want to make a filter, keywords used in CVE only 2015 year.
Thanks!
http://127.0.0.1:5000/cve/CVE-2014-5760
File ".../git/cve-search/web/templates/cve.html", line 148, in <module>
{% for item in k[1]|dictsort %}
File "/usr/local/lib/python3.4/dist-packages/jinja2/filters.py", line 223, in do_dictsort
return sorted(value.items(), key=sort_func)
AttributeError: 'str' object has no attribute 'items'
@PidgeyL Could you reproduce the bug?
Should we integrate all the features from CVE-Search into the API? With this, I mean features like
On a computer with Windows 2008 R2 and running MongoDB OK I want to install CVE-SEARCH
But I'm only interested in the updated database, I do not want the web interface
I just want to download from Sources (eg http://static.nvd.nist.gov/feeds/xml/cve/) the CVE database and I store them in the local database MongoDB on Windows.
I have installed python3
It's possible?
when starting exploitdb, respone below error:
Cannot open url https://github.com/offensive-security/exploit-database/raw/master/files.csv. Bad URL or not connected to the internet?
Check the https://github.com/offensive-security/exploit-database, the files.csv url is https://github.com/offensive-security/exploit-database/blob/master/files.csv
I want to create a JSON, XML or CSV file, but only CVSS more 7 and that has been modified between two dates.
Example:
CVSS> 7
Between:
01/11/2015 and 11/31/2015
Or it would be like searching the web but you need the data in JSON / XML or CSV, or export this result in json or others formats.
I have tried to do with db_dump.py
search.py
search_fulltext.py
But I can not find the correct command
Anyone know?
Thank you!
Traceback (most recent call last):
File "/root/cve-search/sbin/db_mgmt_misp.py", line 16, in <module>
import pytz
ImportError: No module named 'pytz'
First of all, my apologies, I'm not a python developer, so I'm really just trying to run the commands from the README to get the cve-search running.
I'm running on a Mac, I think I the right python (python3 -V / Python 3.4.3
), and I've run the first command (sudo pip3 install...
).
Now from the root of the project directory, I'm running the db_mgmt.py
and it's throwing an error (note that if I run without the sbin
it just can't load the db_mgmt.py
libary):
» python3 ./sbin/db_mgmt.py -p
Traceback (most recent call last):
File "./sbin/db_mgmt.py", line 23, in <module>
from lib.Toolkit import toStringFormattedCPE
File "/Users/remy/Sites/clones/cve-search/sbin/../lib/Toolkit.py", line 11, in <module>
from dateutil import tz
ImportError: No module named 'dateutil'
Anyone able to help?
Hello there!
If it's not a bug, I apologize.
But if I run the command that is written in "brucon2015-cve-search.pdf", page 10
python3 ./bin/search_fulltext.py -q unknown -f | jq -r '. | . vulnerable_configuration[0]' | cut -f5 -d: | sort | uniq -c | sort -nr | head -10
The result is:
4078 }
4078 {
3930 oracle
634 hp
564 google
492 sun
270 mozilla
224 ibm
166 adobe
70 mysql
As you can see, the first two results are {and} when it should not be.
This also occurs in the example on page 11.
python3 ./bin/search_fulltext.py -q unknown -f | jq -r '. | . vulnerable_configuration[0]' | cut -f5,6 -d: | sort | uniq -c | sort -nr | head -10
4078 }
4078 {
504 oracle:database_server
498 google:chrome
388 oracle:fusion_middleware
356 oracle:jre
318 oracle:e-business_suite
298 oracle:mysql
226 oracle:jdk
216 sun:jre
Any idea how to fix it?
Thanks!
Hello
I'm not sure this is the right place to ask about it as it may not be a real issue, but I've trouble with accessing the built-in webserver.
I installed cve-search and in a VM with 2 network adapters. One is configured as NAT, the other is host-only. Apparantly it is not possible to access the VM from the host using port forwarding in NAT: 10.0.2.15:5000 (guest) > 127.0.0.1:5000 (host). Connection via the host-only adapter on 192.168.56.200:5000 doesn't work either.
Is there a setting to specify which IP(s) the built-in server must listens to? I have the impression that it will only accept connections from localhost.
Even with a specific rule to accept connections on port 5000 in iptables, I get the following browser error; "The connection to the server was reset while the page was loading."
Using the host only network's IP, I get "Firefox can't establish a connection to the server at 192.168.10.56:5000."
sudo iptables -A INPUT -p tcp -s 0/0 --sport 1024:65535 -d 192.168.56.200 --dport 5000 -m state --state NEW,ESTABLISHED -j ACCEPT
sudo iptables -A OUTPUT -p tcp -s 192.168.56.200 --sport 5000 -d 0/0 --dport 1024:65535 -m state --state ESTABLISHED -j ACCEPT
Locally on the VM itself, it works only on 127.0.0.1:5000
Using 192.168.56.200:5000 or 10.0.2.15:5000 it doesn't accept connections either.
Thank you in advance for your time.
Hello,
maybe this is my specific issue. On Ubuntu 14.04 looks like full text search doesn't work / doesn't generate full text index.
Running command:
./sbin/db_fulltext.py
Processing[##################################################] 5/5
tooks only about 1 second, and the full text querying seems not to work:
./bin/search_fulltext.py -q NFS -q Linux
(...) no output
How to debug / resolve issue?
Thank you!
Starting vendor
Traceback (most recent call last): ] 0/1440
File ".../cve-search/db_mgmt_vendorstatements.py", line 83, in <module>
bulk.find({'id': statement['id']}).upsert().update({'id': statement['id']}, {"$set":{'statement': statement['statement'], 'id': statement['id'], 'organization': statement['organization'], 'contributor': statement['contributor'], 'lastmodified': statement['lastmodified']}})
TypeError: update() takes 2 positional arguments but 3 were given
As cve-search aggregates various sources of vulnerability, it's not uncommon to get vulnerability without CVE assignment. The idea is to allocate an UUID at the import to ensure a unique identified within
a cve-search installation or beyond cve-search.
The proposal, in a near future, would be to support a public version of cve-search were security researchers could get an UUID for a vulnerability without the need to ask a CVE NA. This UUID could be
used as an unique reference later on even if there is no official CVE assignment or vendor assignment.
Tried to update my db and getting errors
Did a git pull after the first time i saw this and its still getting the same error
python3 db_updater.py -v
Not modified
Not modified
Not modified
Not modified
Not modified
Not modified
Not modified
Not modified
Not modified
[+]Success to create index id on cpe
[+]Success to create index id on cpeother
[+]Success to create index id on cves
[+]Success to create index vulnerable_configuration on cves
[+]Success to create index Modified on cves
[+]Success to create index [('summary', 'text')] on cves
[+]Success to create index id on vfeed
[+]Success to create index id on vendor
[+]Success to create index id on d2sec
[+]Success to create index id on mgmt_whitelist
[+]Success to create index id on mgmt_blacklist
[+]Success to create index related_weakness on capec
Starting cves
cves has 72400 elements (0 update)
Starting cpe
cpe has 106092 elements (0 update)
Starting cpeother
cpeother has 252334 elements (0 update)
Starting vfeed
vfeed has 73146 elements (0 update)
Starting vendor
vendor has 1406 elements (0 update)
Starting cwe
Traceback (most recent call last):
File "/root/cve-search/sbin/db_mgmt_cwe.py", line 107, in <module>
parser.parse(f)
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python3.4/xml/sax/xmlreader.py", line 124, in parse
buffer = file.read(self._bufsize)
File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 46797: ordinal not in range(128)
cwe has 0 elements (0 update)
Starting capec
capec has 463 elements (0 update)
Starting d2sec
d2sec has 246 elements (0 update)
Starting ms
ms has 1345 elements (0 update)
Starting redis-nist-ref
redis-nist-ref has 0 elements (0 update)
Starting ensureindex
New CVEs are frequently added in NVD initially without a CVSS score which then gets populated at a later stage, for example have a look at all the last added CVEs. When querying the local cve-search db for such vulnerabilities it will return a bogus CVSS base score of 5.
# /cve-search/sbin/db_updater.py
[...]
{"Modified": "2015-09-27T22:59:13.090-04:00", "Published": "2015-09-27T22:59:12.013-04:00", "_id": {"$oid": "560874f58864332bfe32fbc6"}, "cvss": 5, "cwe": "Unknown", "id": "CVE-2015-6280", "last-modified": "2015-09-27T22:59:13.090-04:00", "references": ["http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20150923-sshpk/cvrf/cisco-sa-20150923-sshpk_cvrf.xml", "http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20150923-sshpk"], "summary": "The SSHv2 functionality in Cisco IOS 15.2, 15.3, 15.4, and 15.5 and IOS XE 3.6E before 3.6.3E, 3.7E before 3.7.1E, 3.10S before 3.10.6S, 3.11S before 3.11.4S, 3.12S before 3.12.3S, 3.13S before 3.13.3S, and 3.14S before 3.14.1S does not properly implement RSA authentication, which allows remote attackers to obtain login access by leveraging knowledge of a username and the associated public key, aka Bug ID CSCus73013.", "vulnerable_configuration": [], "vulnerable_configuration_cpe_2_2": []}
Note the bogus "cvss": 5
. One way to detect such spurious scores in the DB is checking that the components of the CVSS vector are not present.
Hello,
There's a type conversion error when I'm running "search.py" and outputting the results into a html file.
The issue is on line 159, and the message is "TypeError: Can't convert 'float' object to str implicitly".
I believe the issue can be fixed by changing "item['cvss']" to "str(item['cvss'])".
Cheers.
On several devices already, I noticed that CVE-Search adds entries in the past (several days) without CVSS. I am not sure if this is a CVE-Search issue, an NVD issue, an NVD process update, or just a temporary issue they have, but it would be worth taking a look. Have you noticed something similar, @adulau ?
Should we add OVAL (Open Vulnerability and Assessment Language)information to the database?
(http://oval.mitre.org/)
An example can be found below. Your thoughts?
@adulau @wimremes
<definition id="oval:org.mitre.oval:def:9995" version="5" class="vulnerability">
<metadata>
<title>The Linux kernel before 2.6.16.9 and the FreeBSD kernel, when running on AMD64 and other 7th and 8th generation AuthenticAMD processors, only save/restore the FOP, FIP, and FDP x87 registers in FXSAVE/FXRSTOR when an exception is pending, which allows one process to determine portions of the state of floating point instructions of other processes, which can be leveraged to obtain sensitive information such as cryptographic keys. NOTE: this is the documented behavior of AMD64 processors, but it is inconsistent with Intel processers in a security-relevant fashion that was not addressed by the kernels.</title>
<affected family="unix">
<platform>Red Hat Enterprise Linux 3</platform>
<platform>CentOS Linux 3</platform>
<platform>Red Hat Enterprise Linux 4</platform>
<platform>CentOS Linux 4</platform>
<platform>Oracle Linux 4</platform>
</affected>
<reference source="CVE" ref_id="CVE-2006-1056" ref_url="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-1056"/>
<description>The Linux kernel before 2.6.16.9 and the FreeBSD kernel, when running on AMD64 and other 7th and 8th generation AuthenticAMD processors, only save/restore the FOP, FIP, and FDP x87 registers in FXSAVE/FXRSTOR when an exception is pending, which allows one process to determine portions of the state of floating point instructions of other processes, which can be leveraged to obtain sensitive information such as cryptographic keys. NOTE: this is the documented behavior of AMD64 processors, but it is inconsistent with Intel processers in a security-relevant fashion that was not addressed by the kernels.</description>
<oval_repository>
<dates>
<submitted date="2010-07-09T03:56:16-04:00">
<contributor organization="SCAP.com, LLC">Aharon Chernin</contributor>
</submitted>
<status_change date="2010-07-28T14:25:05.980-04:00">DRAFT</status_change>
<status_change date="2010-08-16T04:15:26.348-04:00">INTERIM</status_change>
<status_change date="2010-09-06T04:16:35.189-04:00">ACCEPTED</status_change>
<modified comment="EDITED oval:org.mitre.oval:def:9995 - Expanded the vulnerability checks for RHEL 3, 4, and 5 to cover CentOS 3, 4, 5 and Oracle Linux 4 and 5" date="2013-04-10T16:24:00.823-04:00">
<contributor organization="G2, Inc.">Dragos Prisaca</contributor>
</modified>
<status_change date="2013-04-10T16:30:25.620-04:00">INTERIM</status_change>
<status_change date="2013-04-29T04:23:59.785-04:00">ACCEPTED</status_change>
</dates>
<status>ACCEPTED</status>
</oval_repository>
</metadata>
<criteria operator="OR">
<criteria operator="AND" comment="OS Section: RHEL3, CentOS3">
<criteria operator="OR" comment="RHEL3 or CentOS3">
<extend_definition comment="The operating system installed on the system is Red Hat Enterprise Linux 3" definition_ref="oval:org.mitre.oval:def:11782"/>
<extend_definition comment="CentOS Linux 3.x" definition_ref="oval:org.mitre.oval:def:16651"/>
</criteria>
<criteria operator="OR" comment="Configuration section">
<criterion comment="kernel-BOOT is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32158"/>
<criterion comment="kernel-unsupported is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32589"/>
<criterion comment="kernel-smp-unsupported is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32704"/>
<criterion comment="kernel-hugemem-unsupported is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32562"/>
<criterion comment="kernel-hugemem is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32078"/>
<criterion comment="kernel is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32513"/>
<criterion comment="kernel-source is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32231"/>
<criterion comment="kernel-doc is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32097"/>
<criterion comment="kernel-smp is earlier than 0:2.4.21-47.EL" test_ref="oval:org.mitre.oval:tst:32708"/>
</criteria>
</criteria>
<criteria operator="AND" comment="OS Section: RHEL4, CentOS4, Oracle Linux 4">
<criteria operator="OR" comment="RHEL4, CentOS4 or Oracle Linux 4">
<extend_definition comment="The operating system installed on the system is Red Hat Enterprise Linux 4" definition_ref="oval:org.mitre.oval:def:11831"/>
<extend_definition comment="CentOS Linux 4.x" definition_ref="oval:org.mitre.oval:def:16636"/>
<extend_definition comment="Oracle Linux 4.x" definition_ref="oval:org.mitre.oval:def:15990"/>
</criteria>
<criteria operator="OR" comment="Configuration section">
<criterion comment="kernel-hugemem is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32335"/>
<criterion comment="kernel-hugemem-devel is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32833"/>
<criterion comment="kernel-smp-devel is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32825"/>
<criterion comment="kernel-largesmp-devel is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32836"/>
<criterion comment="kernel is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32736"/>
<criterion comment="kernel-devel is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:31931"/>
<criterion comment="kernel-doc is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32361"/>
<criterion comment="kernel-largesmp is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32793"/>
<criterion comment="kernel-smp is earlier than 0:2.6.9-42.EL" test_ref="oval:org.mitre.oval:tst:32795"/>
</criteria>
</criteria>
</criteria>
</definition>
Hello! I am having a problem (Ubuntu 14.04 64-bit)
I only happens in index.py
If I run minimal-web.py works OK .
But if I run Index.py, then enters 127.0.0.1:5000 shows me the following error
Will I be able to help ? Thank you!
Error in Web Browser
builtins.TypeError
TypeError: 'bool' object is not callable
Traceback (most recent call last)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/csirt/cve-search/web/index.py", line 273, in index
[Display the sourcecode for this frame] [Open an interactive python shell in this frame] timeTypeSelect, cvssSelect, cvss, rejectedSelect, hideSeen, pageLength, 0)
File "/home/csirt/cve-search/web/index.py", line 177, in filter_logic
if current_user.is_authenticated():
TypeError: 'bool' object is not callable
Error terminal:
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1836, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.4/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/csirt/cve-search/web/index.py", line 273, in index
timeTypeSelect, cvssSelect, cvss, rejectedSelect, hideSeen, pageLength, 0)
File "/home/csirt/cve-search/web/index.py", line 177, in filter_logic
if current_user.is_authenticated():
TypeError: 'bool' object is not callable
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=jquery.js HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
127.0.0.1 - - [17/Sep/2015 13:27:26] "GET /?__debugger__=yes&cmd=resource&f=source.png HTTP/1.1" 200 -
Error Screenshot:
Hello there!
Run the command that is on the sheet 10 PDF brucon2015-cve-search.pdf and it works perfect.
python3 ./bin/search_fulltext.py -q unknown -f | jq -c '. | . vulnerable_configuration[0]' | cut -f5 -d: | sort | uniq -c | sort -nr | head -10
Result:
1965 oracle
317 hp
283 google
246 sun
135 mozilla
116 ibm
83 adobe
35 mysql
31 microsoft
25 novell
But what I want to do is to that command, adding a date search.
I use this command:
python3 ./bin/search_fulltext.py -q unknown -f | jq -c '. 'vulnerable_configuration[0]' and ((.Modified >= "2016-01-01") and (.Modified <= "2016-01-30"))' | cut -f5 -d: | sort | uniq -c | sort -nr | head -10
The result is different from the first command
4078 false
5 true
I knew the command and ((.Modified >= "2016-01-01")
through issue #116
What am I doing wrong?
Gracias!
Hi,
after ProgressBar correction db_mgmt_cpe_other_dictionary.py
goes forward and on line 86 it tries to insert empty batch to
cpeother which causes the error.
I added following test:
if len(batch) != 0:
cpeother.insert(batch)
#update database info after successful program-run
info.update({'db': 'cpeother'}, {"$set": {'last-modified': icve['last-modified']}}, upsert=True)
which fix this issue.
br,
Raino
Hi,
as part of the setup i'm executing db_mgmt_cpe_dictionary.py, which runs for a very long time but doesnt complete. i've let it run for over 3 hours, but still nothing.
when i ctrl-c the execution this is the output.
should i just let it run longer ? or is there something not working here ?
Thanks,
When we populate we try to pull from http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-2002.xml and we are redirected to https://nvd.nist.gov/Data-Feeds/datafeedinfo which says:
Effective October 16, 2015 the XML data feeds will no longer be available for download in an uncompressed format.
We're referred here for the compressed versions.
Received a report from a user having an issue:
python3 ./db_mgmt.py -p
Database population started
Year 2002 imported.
Traceback (most recent call last):
File "./db_mgmt.py", line 186, in <module>
ret = collection.insert(ch.cves)
File "/usr/local/lib/python3.4/site-packages/pymongo/collection.py", line 410, in insert
_check_write_command_response(results)
File "/usr/local/lib/python3.4/site-packages/pymongo/helpers.py", line 198, in _check_write_command_response
raise DuplicateKeyError(error.get("errmsg"), 11000, error)
pymongo.errors.DuplicateKeyError: insertDocument :: caused by :: 11000 E11000 duplicate key error index: cvedb.cves.$_id_ dup key: { : ObjectId('54c7b9d737b9ad76355dcd4b') }
After setup and in doing an update I get the following error:
Starting d2sec
Traceback (most recent call last):
File "/home/ttrostel/cve-search-master/sbin/db_mgmt_d2sec.py", line 109, in
parser.parse(f)
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python3.4/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
File "../Modules/pyexpat.c", line 459, in EndElement
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 307, in end_element
self._cont_handler.endElement(name)
File "/home/ttrostel/cve-search-master/sbin/db_mgmt_d2sec.py", line 74, in endElement
if self.cveref != "":
AttributeError: 'ExploitHandler' object has no attribute 'cveref'
d2sec has 0 elements (0 update)
Any idea what is wrong or how it can be fixed?
Thanks
You're using mongodb.
When cleaning up the code, we were wondering where getBlackRules is defined... Maybe it's a dead part of the code. @PidgeyL Thank you.
./sbin/db_mgmt_d2sec.py threw the following error:
Traceback (most recent call last):
File "/home/PidgeyL/git/PidgeyL/cve-search/sbin/db_mgmt_d2sec.py", line 109, in <module>
parser.parse(f)
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 107, in parse
xmlreader.IncrementalParser.parse(self, source)
File "/usr/lib/python3.4/xml/sax/xmlreader.py", line 123, in parse
self.feed(buffer)
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 207, in feed
self._parser.Parse(data, isFinal)
File "../Modules/pyexpat.c", line 459, in EndElement
File "/usr/lib/python3.4/xml/sax/expatreader.py", line 307, in end_element
self._cont_handler.endElement(name)
File "/home/PidgeyL/git/PidgeyL/cve-search/sbin/db_mgmt_d2sec.py", line 74, in endElement
if self.cveref != "":
AttributeError: 'ExploitHandler' object has no attribute 'cveref'
Hi! have a problem running the command python3 search_fulltext.py -g -s >cve.json
He shows me the following
XXXXX@ubuntu:~/cve-search/bin$ python3 search_fulltext.py -g -s >cve.json
Traceback (most recent call last):
File "search_fulltext.py", line 28, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
I have Whoosh==2.7.0
Is it a bug? Or am I doing something wrong?
Thank you
Not sure if this is an issue or it's just the way it works. I'm trying to use the software (great idea btw, thank you for doing this). I'm kind of stuck on the db_mgmt_cpe_other_dictionary.py script.
It starts off well, running its thing quickly. Then after a few hours the process slows way down, according to the progress bar. It's been running for over 12 hours now.
mongostat
mongostat
connected to: 127.0.0.1
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
*0 *0 *0 *0 0 69|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:20
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:21
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:22
*0 *0 *0 *0 0 72|0 1 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 8k 3 05:22:23
*0 *0 *0 *0 0 69|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:24
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:25
*0 3 *0 *0 0 69|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:26
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:27
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:28
*0 *0 *0 *0 0 71|0 0 1.03g 2.3g 472m 0 cvedb:0.0% 0 0|0 1|0 7k 7k 3 05:22:29
mongotop
connected to: 127.0.0.1
ns total read write 2015-05-28T12:23:09
cvedb.cpe 1967ms 1967ms 0ms
cvedb.cpeother 2ms 2ms 0ms
admin.system.indexes 0ms 0ms 0ms
admin.system.roles 0ms 0ms 0ms
admin.system.users 0ms 0ms 0ms
admin.system.version 0ms 0ms 0ms
ns total read write 2015-05-28T12:23:10
cvedb.cpe 1917ms 1917ms 0ms
cvedb.cpeother 1ms 1ms 0ms
admin.system.indexes 0ms 0ms 0ms
admin.system.roles 0ms 0ms 0ms
admin.system.users 0ms 0ms 0ms
admin.system.version 0ms 0ms 0ms
ns total read write 2015-05-28T12:23:11
cvedb.cpe 1958ms 1958ms 0ms
cvedb.cpeother 1ms 1ms 0ms
admin.system.indexes 0ms 0ms 0ms
admin.system.roles 0ms 0ms 0ms
admin.system.users 0ms 0ms 0ms
admin.system.version 0ms 0ms 0ms
Unix top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
628 mongodb 20 0 2408028 484064 439436 S 97.1 23.7 1153:57 mongod
1311 root 20 0 985416 794980 7664 S 2.7 39.0 22:46.23 python3
api to get last modified (and not published)
Hey folks,
I hope you don't mind me to create a install page, containing instructions on how to setup a linux debian machine to run the project, e.g. which packages must be installed, etc.
Regards
Joao
Hi!
I have a problem.
I Run this command
python3 cve_doc.py -c CVE-2015-0997 | asciidoctor - >test.html
and get an error
Traceback (most recent call last):
File "cve_doc.py", line 47, in <module>
if cve['impact']:
KeyError: 'impact'
But i run a another CVE, work OK (Html perfect)
python3 cve_doc.py -c CVE-2015-0003 | asciidoctor - >test.html
This error is also present in other CVE, for example CVE-2016-0997 and CVE-2016-0997
Why it works only in some CVE?
Greetings.
When the blacklist grows too much, the regex becomes too long for mongo to handle.
Hi!
I have an error when running the command python3 dump_last.py -r -l 20 -f html
XXX@XXX:~/cve-search/bin$ python3 dump_last.py -r -l 20 -f html
<html><head>
<style>.cve table { border-collapse: collapse; text-align: left; width: 100%; } .cve {font: normal 12px/150% Geneva, Arial, Helvetica, sans-serif; background: #fff; overflow: hidden; border: 1px solid #006699; -webkit-border-radius: 3px; -moz-border-radius: 3px; border-radius: 3px; }.cve table td, .cve table th { padding: 3px 10px; }.cve table tbody td { color: #00496B; border-left: 1px solid #E1EEF4;font-size: 12px;font-weight: normal; }.cve table tbody .alt td { background: #E1EEF4; color: #00496B; }.cve table tbody td:first-child { border-left: none; }.cve table tbody tr:last-child td { border-bottom: none; }.cve table tfoot td div { border-top: 1px solid #006699;background: #E1EEF4;} .cve table tfoot td { padding: 0; font-size: 12px } .cve table tfoot td div{ padding: 0px; }</style>
<title>Last 20 CVE entries</title>
</head><body>
Traceback (most recent call last):
File "dump_last.py", line 72, in <module>
if not x['ranking']:
KeyError: 'ranking'
When I add > test.html
XXX@XXX:~/cve-search/bin$ python3 dump_last.py -r -l 20 -f html > test.html
Traceback (most recent call last):
File "dump_last.py", line 72, in <module>
if not x['ranking']:
KeyError: 'ranking'
The html generated is blank but watch the source code
<html><head>
<style>.cve table { border-collapse: collapse; text-align: left; width: 100%; } .cve {font: normal 12px/150% Geneva, Arial, Helvetica, sans-serif; background: #fff; overflow: hidden; border: 1px solid #006699; -webkit-border-radius: 3px; -moz-border-radius: 3px; border-radius: 3px; }.cve table td, .cve table th { padding: 3px 10px; }.cve table tbody td { color: #00496B; border-left: 1px solid #E1EEF4;font-size: 12px;font-weight: normal; }.cve table tbody .alt td { background: #E1EEF4; color: #00496B; }.cve table tbody td:first-child { border-left: none; }.cve table tbody tr:last-child td { border-bottom: none; }.cve table tfoot td div { border-top: 1px solid #006699;background: #E1EEF4;} .cve table tfoot td { padding: 0; font-size: 12px } .cve table tfoot td div{ padding: 0px; }</style>
<title>Last 20 CVE entries</title>
</head><body>
If I run the command python3 dump_last.py -r -l 20 -f rss1 > test.xml
I generates XML that contains only this:
<rdf:RDF xmlns="http://purl.org/rss/1.0/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"><channel rdf:about="http://adulau.github.com/cve-search/"><title>cve-search Last 20 CVE entries generated on 2015-12-17 12:08:46.866382</title><link>http://adulau.github.com/cve-search/</link><description /><items><rdf:Seq /></items></channel></rdf:RDF>
The command I've seen in
"https://github.com/cve-search/cve-search/blob/master/doc/talks/brucon2015-cve-search.pdf"
page 14
Adding an option to not have any admin interface or write-access to the database on the web interface.
Two updates need to be done:
What do you think @PidgeyL ?
Update database (pyhton3 db_updater.py) fails on Windows 2008 R2 SP1, Python 3.5.1
pyhton3 db_mgmt.py -p (OK)
pyhton3 db_mgmt_cpe_dictionary.py (OK)
pyhton3 db_updater.py -c (FAIL)
C:\cve-search-master\sbin>python db_updater.py -h Traceback (most recent call last): File "db_updater.py", line 17, in <module> import syslog ImportError: No module named 'syslog'
I'm not sure, but ...
Can it be like this?
arista-eosplus/pyeapi#10
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.