joshbressers / cve-analysis Goto Github PK

Hi,

I had a task of putting the nvds inside ES and used your project to inspire me.

Since there were so many files, I wrote a single python file that does the same functions as some files.

The script I created encompasses the functions of the following scripts:
-get-cve-json.sh
-get-cve.sh
-json-parse.py
-update-es.sh

are you interested in PR?
you can find the code bellow:

https://github.com/lucassoccol/cve-analysis/blob/master/cve_to_es.py

Hi,

As per subject, I am having problem importing cve-kibana.ndjson
I have tried Elasticsearch/Kibana 7.3.1 and 7.0.1, after clicking Import I get

Sorry, there was an error
The file could not be processed.

I came across this article, claiming that warning is triggered by JSON parser
https://discuss.elastic.co/t/kibana-dashboard-import-failed-this-file-could-not-be-processed/91181

when I tried to validate cve-kibana.ndjson using https://jsonformatter.org/ I got following:
Parse error on line 1:
...rsion":"WzEyLDNd"}
{"attributes":{"desc
----------------------^
Expecting 'EOF', '}', ',', ']', got '{'

Has anybody experienced similar issue ??

Thanks

Traceback (most recent call last):
File "./json-parse.py", line 31, in
main()
File "./json-parse.py", line 28, in main
:cve, 'doc_as_upsert': True})
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/init.py", line 661, in update
"POST", _make_path(index, doc_type, id, "_update"), params=params, body=body
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 318, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 239, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 131, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'failed to parse field [description] of type [text]')

I get the following error when trying to use the curl command from the "Quick start guide". Not being an expert on elasticsearch I thought you might be able to tell me if I am doing something wrong or if it's a compatability issue with elasticsearch 7x? :

sudo curl -XPUT 'localhost:9200/cve-index?pretty' -H 'Content-Type: application/json' -d @cve-index.json

{ "error" : { "root_cause" : [ { "type" : "mapper_parsing_exception", "reason" : "Root mapping definition has unsupported parameters: [doc : {properties={fromNVD={type=long}, cvss:confidentiality-impact={type=text, fields={keyword={type=keyword}}}, product={type=text, fields={keyword={type=keyword}}}, cvss:generated-on-datetime={type=date}, year={type=long}, cvss:source={type=text, fields={keyword={type=keyword}}}, cvss:score={type=long}, description={fielddata=true, analyzer=english, type=text, fields={shingle={fielddata=true, analyzer=analyzer_shingle, type=text}, keyword={type=keyword}}}, cvss:authentication={type=text, fields={keyword={type=keyword}}}, cvss:access-vector={type=text, fields={keyword={type=keyword}}}, cwe={type=text, fields={keyword={type=keyword}}}, cvss:integrity-impact={type=text, fields={keyword={type=keyword}}}, fromCVE={type=long}, id={type=long}, cvss:availability-impact={type=text, fields={keyword={type=keyword}}}, cvss:access-complexity={type=text, fields={keyword={type=keyword}}}}}]" } ], "type" : "mapper_parsing_exception", "reason" : "Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [doc : {properties={fromNVD={type=long}, cvss:confidentiality-impact={type=text, fields={keyword={type=keyword}}}, product={type=text, fields={keyword={type=keyword}}}, cvss:generated-on-datetime={type=date}, year={type=long}, cvss:source={type=text, fields={keyword={type=keyword}}}, cvss:score={type=long}, description={fielddata=true, analyzer=english, type=text, fields={shingle={fielddata=true, analyzer=analyzer_shingle, type=text}, keyword={type=keyword}}}, cvss:authentication={type=text, fields={keyword={type=keyword}}}, cvss:access-vector={type=text, fields={keyword={type=keyword}}}, cwe={type=text, fields={keyword={type=keyword}}}, cvss:integrity-impact={type=text, fields={keyword={type=keyword}}}, fromCVE={type=long}, id={type=long}, cvss:availability-impact={type=text, fields={keyword={type=keyword}}}, cvss:access-complexity={type=text, fields={keyword={type=keyword}}}}}]", "caused_by" : { "type" : "mapper_parsing_exception", "reason" : "Root mapping definition has unsupported parameters: [doc : {properties={fromNVD={type=long}, cvss:confidentiality-impact={type=text, fields={keyword={type=keyword}}}, product={type=text, fields={keyword={type=keyword}}}, cvss:generated-on-datetime={type=date}, year={type=long}, cvss:source={type=text, fields={keyword={type=keyword}}}, cvss:score={type=long}, description={fielddata=true, analyzer=english, type=text, fields={shingle={fielddata=true, analyzer=analyzer_shingle, type=text}, keyword={type=keyword}}}, cvss:authentication={type=text, fields={keyword={type=keyword}}}, cvss:access-vector={type=text, fields={keyword={type=keyword}}}, cwe={type=text, fields={keyword={type=keyword}}}, cvss:integrity-impact={type=text, fields={keyword={type=keyword}}}, fromCVE={type=long}, id={type=long}, cvss:availability-impact={type=text, fields={keyword={type=keyword}}}, cvss:access-complexity={type=text, fields={keyword={type=keyword}}}}}]" } }, "status" : 400 }

Description tagcloud, CVSSv3 Histogram and CVSSv2 Histogram display error

when you hover over Description tagcloud you get the follow pop up [esaggs]> Saved field "description.description_value" of index pattern "cve-index" is invalid for use with the "Terms" aggregation. Please select a new field.

when you hover over CVSSv3 Histogram you get the follow pop up [esagg]> Unable to retrieve max and min values to auto-scale histogram buckets. This may lead to poor visualization performance.

when you hover over CVSSv2 Histogram you get the follow pop up Unable to retrieve max and min values to auto-scale histogram buckets. This may lead to poor visualization performance.

Any idea what is going on with this?

Thanks

Loading JSON data/nvdcve-1.0-2017.json
POST http://localhost:9200/cve-index/doc/CVE-2017-0001/_update [status:400 request:0.016s]
Traceback (most recent call last):
File "./json-parse.py", line 30, in
main()
File "./json-parse.py", line 27, in main
es.update(id=cve_id, index="cve-index", doc_type='doc', body={'doc':cve, 'doc_as_upsert': True})
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/client/init.py", line 547, in update
doc_type, id, '_update'), params=params, body=body)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/transport.py", line 318, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/http_urllib3.py", line 186, in perform_request
self._raise_error(response.status, raw_data)
File "/usr/local/lib/python3.6/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.RequestError: RequestError(400, 'mapper_parsing_exception', 'failed to parse field [description] of type [text]')

Hello,

I want to try your CVE analysis.
At the end of the README, it is recommended to import cve-dashboard.json and cve-visualizations.json but these files have been deleted in your last push.
how to do it?

Thanks for your answers !

Aymeric Nosjean
[email protected]

Hi, I want to include the configurations part in the mapping in order to have also the information about the cpe fram the CVE. Furthermore, something goes wrong and once I load the CVEs on Elastic, they have not those fields.

in json-parse.py you specify the server as
if 'ESURL' not in os.environ:
es_url = "http://localhost:9200"

What would be the correct syntax for https? with a self signed certificate?
I have tried
es_url = "https://localhost:9200", ca_certs="/path/to/http_ca.crt"

But all this does is fail saying that it is looking for a list of hosts.

Hi,
I get the follwoing issue when I am running the script update-es.sh

Loading JSON data/nvdcve-1.1-2003.json and all others
...
import elasticsearch
ModuleNotFoundError: No module named 'elasticsearch'

I use Debian 10 actual iso with the installation guide form
https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elastic-stack-on-ubuntu-18-04