fabric8-analytics / fabric8-analytics-data-model Goto Github PK
View Code? Open in Web Editor NEWLicense: GNU General Public License v3.0
fabric8-analytics-data-model's People
Contributors
Stargazers
Watchers
Forkers
fridex jpopelka pombredanne miteshvp saketjnu msrb arunkumars08 concaf tuxdna tisnik ravsa sara-02 abs51295 sunilk747 mldeveloper01 hrishin geetikabatra jmelis yzainee-zz humaton cnulenka avni-sharma samuzzal-choudhury animuk arajkumar dvandra deepak1725 rajusem jparsai practice-fabric8-analytics sawood14012 rafiu007 divyansh42 vinagarw272001fabric8-analytics-data-model's Issues
No such bucket error encountered
Start the import :
http://localhost:32000/api/v1/component-analyses/maven/org.apache.tomcat:tomcat-servlet-api/8.5.6
Data import into graph fails for a package: /maven/org.apache.tomcat:tomcat-servlet-api/8.5.6
data-model-importer_1 | --------------------------------------------------------------------------------
data-model-importer_1 | INFO in rest_api [/src/rest_api.py:94]:
data-model-importer_1 | Ingesting the given list of EPVs - [{"ecosystem": "maven", "version": "8.5.6", "name": "org.apache.tomcat:tomcat-servlet-api"}]
data-model-importer_1 | --------------------------------------------------------------------------------
data-model-importer_1 | ERROR:data_importer:import_epv() failed with error: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist
data-model-importer_1 | ERROR:data_importer:Traceback for latest failure in import call: Traceback (most recent call last):
data-model-importer_1 | File "/src/data_importer.py", line 164, in import_epv_http
data-model-importer_1 | prefix=pkg_key_prefix))
data-model-importer_1 | File "/src/data_source/s3_data_source.py", line 66, in list_files
data-model-importer_1 | for obj in bucket.objects.filter(Prefix=prefix):
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/boto3/resources/collection.py", line 83, in __iter__
data-model-importer_1 | for page in self.pages():
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/boto3/resources/collection.py", line 166, in pages
data-model-importer_1 | for page in pages:
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/botocore/paginate.py", line 255, in __iter__
data-model-importer_1 | response = self._make_request(current_kwargs)
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/botocore/paginate.py", line 332, in _make_request
data-model-importer_1 | return self._method(**current_kwargs)
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/botocore/client.py", line 317, in _api_call
data-model-importer_1 | return self._make_api_call(operation_name, kwargs)
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/botocore/client.py", line 615, in _make_api_call
data-model-importer_1 | raise error_class(parsed_response, operation_name)
data-model-importer_1 | NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist
data-model-importer_1 | [2018-01-31 11:29:48 +0000] [70] [ERROR] Error handling request /api/v1/ingest_to_graph
data-model-importer_1 | Traceback (most recent call last):
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
data-model-importer_1 | self.handle_request(listener_name, req, client, addr)
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
data-model-importer_1 | super(GeventWorker, self).handle_request(*args)
data-model-importer_1 | File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
data-model-importer_1 | respiter = self.wsgi(environ, resp.start_response)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
data-model-importer_1 | return self.wsgi_app(environ, start_response)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
data-model-importer_1 | response = self.handle_exception(e)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
data-model-importer_1 | return cors_after_request(app.make_response(f(*args, **kwargs)))
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
data-model-importer_1 | reraise(exc_type, exc_value, tb)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
data-model-importer_1 | response = self.full_dispatch_request()
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
data-model-importer_1 | rv = self.handle_user_exception(e)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
data-model-importer_1 | return cors_after_request(app.make_response(f(*args, **kwargs)))
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
data-model-importer_1 | reraise(exc_type, exc_value, tb)
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
data-model-importer_1 | rv = self.dispatch_request()
data-model-importer_1 | File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
data-model-importer_1 | return self.view_functions[rule.endpoint](**req.view_args)
data-model-importer_1 | File "/src/rest_api.py", line 102, in ingest_to_graph
data-model-importer_1 | report = data_importer.import_epv_from_s3_http(list_epv=input_json)
data-model-importer_1 | File "/src/data_importer.py", line 214, in import_epv_from_s3_http
data-model-importer_1 | list_epv, select_doc)
data-model-importer_1 | File "/src/data_importer.py", line 199, in import_epv_http
data-model-importer_1 | raise RuntimeError(msg)
data-model-importer_1 | RuntimeError: import_epv() failed with error: An error occurred (NoSuchBucket) when calling the ListObjects operation: The specified bucket does not exist
data-model-importer_1 | --------------------------------------------------------------------------------
data-model-importer_1 | INFO in rest_api [/src/rest_api.py:94]:
data-model-importer_1 | Ingesting the given list of EPVs - [{"ecosystem": "maven", "version": null, "name": "org.apache.tomcat:tomcat-servlet-api"}]
Fails to import packages where declared license is multiline
Originally found here: openshiftio/openshift.io#2048 (comment)
Some intermittent failures encountered in Graph layer. This seems to be happening very sparsely:
g.V().has('ecosystem','maven').has('name','org.apache.tomcat:tomcat-servlet-api').properties('tokens','libio_usedby').drop().iterate();pkg = g.V().has('ecosystem','maven').has('name', 'org.apache.tomcat:tomcat-servlet-api').tryNext().orElseGet{graph.addVertex('ecosystem', 'maven', 'name', 'org.apache.tomcat:tomcat-servlet-api', 'vertex_label', 'Package')};pkg.property('last_updated', 1517286212.43);pkg.property('tokens', 'org'); pkg.property('tokens', 'apache'); pkg.property('tokens', 'tomcat'); pkg.property('tokens', 'tomcat'); pkg.property('tokens', 'servlet'); pkg.property('tokens', 'api');pkg.property('latest_version', '9.0.0.M17');pkg.property('libio_latest_release', '1500854400.0');pkg.property('libio_usedby', 'keycloak/keycloak:1359');pkg.property('libio_usedby', 'SungardAS/enhanced-snapshots:35');pkg.property('libio_usedby', 'cf-unik/unik:1239');pkg.property('libio_usedby', 'entando/entando-components:15');pkg.property('libio_usedby', 'indeedeng/proctor:198');pkg.property('libio_usedby', 'magro/memcached-session-manager:552');pkg.property('libio_usedby', 'nysenate/OpenLegislation:163');pkg.property('libio_usedby', 'Red5/red5-server:1254');pkg.property('libio_usedby', 'aspose-words/Aspose.Words-for-Java:52');pkg.property('libio_usedby', 'google/identity-toolkit-java-client:32');pkg.property('libio_dependents_projects', '65');pkg.property('libio_dependents_repos', '2.05K');pkg.property('libio_total_releases', '124');pkg.property('libio_latest_version', '8.5.19');g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.19').property('gh_release_date', 1500854400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M25').property('gh_release_date',1500854400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M19').property('gh_release_date',1490572800.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.
5.16').property('gh_release_date',1498003200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.14').property('gh_release_date',1492041600.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.15').property('gh_release_date',1493942400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M21').property('gh_release_date',1493856000.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M20').property('gh_release_date',1491955200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M22').property('gh_release_date',1498003200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.13').property('gh_release_date',1490572800.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M1').properties('licenses','cve_ids','declared_licenses').drop().iterate();ver = g.V().has('pecosystem', 'maven').has('pname', 'org.apache.tomcat:tomcat-servlet-api').has('version', '9.0.0.M1').tryNext().orElseGet{graph.addVertex('pecosystem','maven', 'pname','org.apache.tomcat:tomcat-servlet-api', 'version', '9.0.0.M1', 'vertex_label', 'Version')};ver.property('last_updated',1517286212.43);ver.property('description','javax.servlet package');ver.property('cm_num_files',112);ver.property('cm_avg_cyclomatic_complexity', 1.23);ver.property('cm_loc',42622);ver.property('licenses', 'ASL 2.0'); ver.property('licenses', 'CDDL');ver.property('cve_ids', 'CVE-2017-6056:5.0'); ver.property('cve_ids', 'CVE-2016-8735:7.5'); ver.property('cve_ids', 'CVE-2016-6816:6.8'); ver.property('cve_ids', 'CVE-2016-6325:7.2'); ver.property('cve_ids', 'CVE-2016-5425:7.2'); ver.property('cve_ids', 'CVE-2016-3092:7.8'); ver.property('cve_ids', 'CVE-2016-0763:6.5'); ver.property('cve_ids', 'CVE-2016-0714:6.
5'); ver.property('cve_ids', 'CVE-2016-0706:4.0'); ver.property('cve_ids', 'CVE-2015-5351:6.8'); ver.property('cve_ids', 'CVE-2015-5346:6.8'); ver.property('cve_ids', 'CVE-2015-5345:5.0');ver.property('declared_licenses', 'Apache License'); ver.property('declared_licenses', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0');lic = g.V().has('lname', 'Apache License').tryNext().orElseGet{graph.addVertex('vertex_label', 'License', 'lname', 'Apache License', 'last_updated',1517286212.43)}; g.V(ver).out('has_declared_license').has('lname', 'Apache License').tryNext().orElseGet{ver.addEdge('has_declared_license', lic)};lic = g.V().has('lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0').tryNext().orElseGet{graph.addVertex('vertex_label', 'License', 'lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0', 'last_updated',1517286212.43)}; g.V(ver).out('has_declared_license').has('lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0').tryNext().orElseGet{ver.addEdge('has_declared_license', lic)};edge_c = g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M1').in('has_version').tryNext().orElseGet{pkg.addEdge('has_version', ver)};
ERROR:data_importer:The import failed: 'status'
ERROR:data_importer:Traceback for latest failure in import call: Traceback (most recent call last):
File "/src/data_importer.py", line 101, in _import_keys_from_s3_http
if resp['status']['code'] == 200:
KeyError: 'status'
g.V().has('ecosystem','maven').has('name','org.apache.tomcat:tomcat-servlet-api').properties('tokens','libio_usedby').drop().iterate();pkg = g.V().has('ecosystem','maven').has('name', 'org.apache.tomcat:tomcat-servlet-api').tryNext().orElseGet{graph.addVertex('ecosystem', 'maven', 'name', 'org.apache.tomcat:tomcat-servlet-api', 'vertex_label', 'Package')};pkg.property('last_updated', 1517286213.63);pkg.property('tokens', 'org'); pkg.property('tokens', 'apache'); pkg.property('tokens', 'tomcat'); pkg.property('tokens', 'tomcat'); pkg.property('tokens', 'servlet'); pkg.property('tokens', 'api');pkg.property('latest_version', '9.0.0.M17');pkg.property('libio_latest_release', '1500854400.0');pkg.property('libio_usedby', 'keycloak/keycloak:1359');pkg.property('libio_usedby', 'SungardAS/enhanced-snapshots:35');pkg.property('libio_usedby', 'cf-unik/unik:1239');pkg.property('libio_usedby', 'entando/entando-components:15');pkg.property('libio_usedby', 'indeedeng/proctor:198');pkg.property('libio_usedby', 'magro/memcached-session-manager:552');pkg.property('libio_usedby', 'nysenate/OpenLegislation:163');pkg.property('libio_usedby', 'Red5/red5-server:1254');pkg.property('libio_usedby', 'aspose-words/Aspose.Words-for-Java:52');pkg.property('libio_usedby', 'google/identity-toolkit-java-client:32');pkg.property('libio_dependents_projects', '65');pkg.property('libio_dependents_repos', '2.05K');pkg.property('libio_total_releases', '124');pkg.property('libio_latest_version', '8.5.19');g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.19').property('gh_release_date', 1500854400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M25').property('gh_release_date',1500854400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M19').property('gh_release_date',1490572800.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.
5.16').property('gh_release_date',1498003200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.14').property('gh_release_date',1492041600.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.15').property('gh_release_date',1493942400.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M21').property('gh_release_date',1493856000.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M20').property('gh_release_date',1491955200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M22').property('gh_release_date',1498003200.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.13').property('gh_release_date',1490572800.0);g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M11').properties('licenses','cve_ids','declared_licenses').drop().iterate();ver = g.V().has('pecosystem', 'maven').has('pname', 'org.apache.tomcat:tomcat-servlet-api').has('version', '9.0.0.M11').tryNext().orElseGet{graph.addVertex('pecosystem','maven', 'pname','org.apache.tomcat:tomcat-servlet-api', 'version', '9.0.0.M11', 'vertex_label', 'Version')};ver.property('last_updated',1517286213.63);ver.property('description','javax.servlet package');ver.property('cm_num_files',114);ver.property('cm_avg_cyclomatic_complexity', 1.23);ver.property('cm_loc',42800);ver.property('licenses', 'ASL 2.0'); ver.property('licenses', 'CDDL');ver.property('cve_ids', 'CVE-2017-6056:5.0'); ver.property('cve_ids', 'CVE-2016-8747:5.0'); ver.property('cve_ids', 'CVE-2016-8735:7.5'); ver.property('cve_ids', 'CVE-2016-6816:6.8'); ver.property('cve_ids', 'CVE-2016-6325:7.2'); ver.property('cve_ids', 'CVE-2016-5425:7.2');ver.property('declared_licenses', 'Apache License'); ver.property('declared_licenses'
, ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0');lic = g.V().has('lname', 'Apache License').tryNext().orElseGet{graph.addVertex('vertex_label', 'License', 'lname', 'Apache License', 'last_updated',1517286213.63)}; g.V(ver).out('has_declared_license').has('lname', 'Apache License').tryNext().orElseGet{ver.addEdge('has_declared_license', lic)};lic = g.V().has('lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0').tryNext().orElseGet{graph.addVertex('vertex_label', 'License', 'lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0', 'last_updated',1517286213.63)}; g.V(ver).out('has_declared_license').has('lname', ' Version 2.0 and
Common Development And Distribution License (CDDL) Version 1.0').tryNext().orElseGet{ver.addEdge('has_declared_license', lic)};edge_c = g.V().has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','9.0.0.M11').in('has_version').tryNext().orElseGet{pkg.addEdge('has_version', ver)};
ERROR:data_importer:The import failed: HTTPConnectionPool(host='172.30.80.86', port=8182): Read timed out. (read timeout=30)
ERROR:data_importer:Traceback for latest failure in import call: Traceback (most recent call last):
File "/src/data_importer.py", line 98, in _import_keys_from_s3_http
data=json.dumps(payload), timeout=30)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 521, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPConnectionPool(host='172.30.80.86', port=8182): Read timed out. (read timeout=30)
Figured out the cause of above error from gremlin server logs:
39989555 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
Script3023.groovy: 1: expecting ''', found '\n' @ line 1, column 4116.
d_licenses', ' Version 2.0 and
^
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script3023.groovy: 1: expecting ''', found '\n' @ line 1, column 4116.
d_licenses', ' Version 2.0 and
^
1 error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:140)
at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Essentially the queries are not formed should factor in special characters like newlines correctly in existing code here:
drop_props.append('declared_licenses')
prp_version += " ".join(["ver.property('declared_licenses', '{}');".format
(dl) for dl in declared_licenses])
# Create License Node and edge from EPV
for lic in declared_licenses:
prp_version += "lic = g.V().has('lname', '{lic}').tryNext().orElseGet{{" \
"graph.addVertex('vertex_label', 'License', 'lname', '{lic}', " \
"'last_updated',{last_updated})}}; g.V(ver).out(" \
"'has_declared_license').has('lname', '{lic}').tryNext()." \
"orElseGet{{ver.addEdge('has_declared_license', lic)}};".format(
lic=lic, last_updated=str(time.time())
)Happens for this package
has('pecosystem','maven').has('pname','org.apache.tomcat:tomcat-servlet-api').has('version','8.5.14')
metadata.json for the above EPV contains
"declared_license": "Apache License, Version 2.0 and\n Common Development And Distribution License (CDDL) Version 1.0",
Notice the newline.
JsonMappingException: Vertex with id 24985604280 was removed
This is an error from logs of instance running in production:
22086511 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=300a8de6-d7ea-4aa5-8216-826bb3a5b57d, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
--
| 22086511 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Error during serialization for ResponseMessage{requestId=300a8de6-d7ea-4aa5-8216-826bb3a5b57d, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}
| org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)
| at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
| at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| Caused by: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
| at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
| at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:311)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:368)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
| ... 7 more
| Caused by: java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
| at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
| at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
| at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
| ... 17 more
| 22086513 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Vertex with id 24985604280 was removed.
| java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
| at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
| at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
| at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
| at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
| at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| 22087866 [gremlin-server-exec-3] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=a419bf20-2d55-4776-ba2c-f70ad93aeecb, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
| 22087866 [gremlin-server-exec-3] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Error during serialization for ResponseMessage{requestId=a419bf20-2d55-4776-ba2c-f70ad93aeecb, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}
| org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)
| at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
| at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| Caused by: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
| at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
| at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:311)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:368)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
| ... 7 more
| Caused by: java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
| at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
| at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
| at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
| ... 17 more
| 22087867 [gremlin-server-exec-3] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Vertex with id 24985604280 was removed.
| java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
| at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
| at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
| at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
| at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
| at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
| at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
| at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
| at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
| at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
| at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
| at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| 22089179 [gremlin-server-exec-6] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=540ad248-41f5-4921-a45c-851965343c99, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
TODO: reduce cyclomatic complexity
This code branch in function get_qstring_for_cve_node needs to be covered by unit test
This code branch in function get_qstring_for_cve_node needs to be covered by unit test
fabric8-analytics-data-model/src/cve.py
Line 95 in 7d93d83
Single quotes break the queries again
As encountered in logs from production instance
44242668 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
| Script4829.groovy: 1: expecting ')', found 'Reilly' @ line 1, column 10197.
| ; ver.property('licenses', 'O'Reilly Cod
| ^
|
| 1 error
|
| org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
| Script4829.groovy: 1: expecting ')', found 'Reilly' @ line 1, column 10197.
| ; ver.property('licenses', 'O'Reilly Cod
| ^
JsonMappingException: Vertex with id SOME_ID_HERE was removed
Not sure what is causing the following error in the logs, so adding it for tracking:
967823 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=d67ac739-6ef4-4685-9cf5-5a9fa29762dd, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[33587556368]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
967824 [gremlin-server-exec-5] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Error during serialization for ResponseMessage{requestId=d67ac739-6ef4-4685-9cf5-5a9fa29762dd, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[33587556368]], meta={}}}
org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 33587556368 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 33587556368 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:311)
at org.apache.tinkerpop.shaded.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:368)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
... 7 more
Caused by: java.lang.IllegalStateException: Vertex with id 33587556368 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
... 17 more
967832 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Vertex with id 33587556368 was removed.
java.lang.IllegalStateException: Vertex with id 33587556368 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8592251 [gremlin-server-exec-8] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=214fb8d8-1ede-423f-ae88-8c66cd5a5a45, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
8592251 [gremlin-server-exec-8] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Error during serialization for ResponseMessage{requestId=214fb8d8-1ede-423f-ae88-8c66cd5a5a45, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}
org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:311)
at org.apache.tinkerpop.shaded.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:368)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
... 7 more
Caused by: java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
... 17 more
8592253 [gremlin-server-exec-8] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Vertex with id 24985604280 was removed.
java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8593721 [gremlin-server-exec-2] WARN org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 - Response [ResponseMessage{requestId=31f0bbca-2b40-4001-90a3-2a5df894b4b9, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}] could not be serialized by org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0.
8593721 [gremlin-server-exec-2] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Error during serialization for ResponseMessage{requestId=31f0bbca-2b40-4001-90a3-2a5df894b4b9, status=ResponseStatus{code=SUCCESS, message='', attributes={}}, result=ResponseResult{data=[v[24985604280]], meta={}}}
org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:101)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException: Vertex with id 24985604280 was removed. (through reference chain: java.util.ArrayList[0])
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:339)
at org.apache.tinkerpop.shaded.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:311)
at org.apache.tinkerpop.shaded.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:368)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
... 7 more
Caused by: java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
... 17 more
8593723 [gremlin-server-exec-2] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Vertex with id 24985604280 was removed.
java.lang.IllegalStateException: Vertex with id 24985604280 was removed.
at org.apache.tinkerpop.gremlin.structure.Element$Exceptions.elementAlreadyRemoved(Element.java:154)
at com.thinkaurelius.titan.core.InvalidElementException.removedException(InvalidElementException.java:44)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.it(AbstractVertex.java:40)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.query(StandardTitanTx.java:1004)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.getVertexLabelInternal(AbstractVertex.java:108)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.vertexLabel(AbstractVertex.java:113)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.label(AbstractVertex.java:104)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.ser(GraphSONSerializersV1d0.java:198)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:182)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONSerializersV1d0$VertexJacksonSerializer.serialize(GraphSONSerializersV1d0.java:170)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
at org.apache.tinkerpop.shaded.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
at org.apache.tinkerpop.gremlin.structure.io.graphson.GraphSONUtil.writeWithType(GraphSONUtil.java:55)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.ser(AbstractGraphSONMessageSerializerV1d0.java:244)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:214)
at org.apache.tinkerpop.gremlin.driver.ser.AbstractGraphSONMessageSerializerV1d0$ResponseMessageSerializer.serialize(AbstractGraphSONMessageSerializerV1d0.java:206)
at org.apache.tinkerpop.shaded.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:130)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3613)
at org.apache.tinkerpop.shaded.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:2980)
at org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0.serializeResponseAsString(GraphSONMessageSerializerV1d0.java:98)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:248)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Quotes causing Gremlin queries to fail
From the following logs it appears that the query is failing due to single quote present in the input data:
8629825 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
Script936.groovy: 1: expecting ')', found 'The' @ line 1, column 1146.
operty('declared_licenses', ''The Apache
^
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script936.groovy: 1: expecting ')', found 'The' @ line 1, column 1146.
operty('declared_licenses', ''The Apache
^
1 error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:145)
at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Queries are not formed properly
Encountered this error in the logs
5340448 [gremlin-server-exec-2] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
Script782.groovy: 1: unexpected char: '\' @ line 1, column 7263.
mber value using Number.equals\ boolean
^
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script782.groovy: 1: unexpected char: '\' @ line 1, column 7263.
mber value using Number.equals\ boolean
^
1 error
at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:140)
at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5341861 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
Script783.groovy: 1: unexpected char: '\' @ line 1, column 7284.
mber value using Number.equals\ boolean
It is likely that some string parameter is not being escaped properly.
Document the usage of Codecov.io for this repository
We use Codecov.io to track code coverage changes. It would be good to document the usage of Codecov.io in the README.md as well.
The import failed: list index out of range
INFO in rest_api [/src/rest_api.py:128]:
--
| Selective Ingestion with payload - {"select_ingest": ["keywords_tagging", "git_stats", "github_details", "libraries_io"], "package_list": [{"ecosystem": "maven", "version": null, "name": "com.yugabyte:cassandra-driver-extras"}]}
| --------------------------------------------------------------------------------
| ERROR:data_importer:The import failed: list index out of range
| ERROR:data_importer:Traceback for latest failure in import call: Traceback (most recent call last):
| File "/src/data_importer.py", line 87, in _import_keys_from_s3_http
| str_gremlin = GraphPopulator.create_query_string(obj)
| File "/src/graph_populator.py", line 361, in create_query_string
| str_package, prp_package = cls.construct_package_query(input_json)
| File "/src/graph_populator.py", line 272, in construct_package_query
| libio_latest = releases.get('recent', [{}])[-1] # last is latest
| IndexError: list index out of range
|
| ERROR:data_importer:
| Report from import_epv():
| The import failed: list index out of range
| Total number of EPVs imported: 0
| The last successfully imported EPV: None
|
| --------------------------------------------------------------------------------
| INFO in rest_api [/src/rest_api.py:136]:
| {'epv': {u'select_ingest': [u'keywords_tagging', u'git_stats', u'github_details', u'libraries_io'], u'package_list': [{u'ecosystem': u'maven', u'version': '', u'name': u'com.yugabyte:cassandra-driver-extras'}]}, 'message': 'The import failed: list index out of range', 'count_imported_EPVs': 0}
| --------------------------------------------------------------------------------
Warning: please use "pip install psycopg2-binary"
This is encountered in Production instance of data-model importer:
/usr/lib64/python2.7/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>;.
--
| """)
Invalid request - responding with 500 Internal Server Error
18093838 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
--
| Script3654.groovy: 1: unexpected char: '\' @ line 1, column 589.
| rces using WebLab properties./\ At the t
| ^
|
| 1 error
|
| org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
| Script3654.groovy: 1: unexpected char: '\' @ line 1, column 589.
| rces using WebLab properties./\ At the t
| ^
|
| 1 error
|
| at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
| at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
| at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:140)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
| at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
| at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
| at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
| at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
| at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
| at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
| at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
| at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
| at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
Seems like it is happening for description field: http://central.maven.org/maven2/org/ow2/weblab/core/annotator/1.2.4.1/annotator-1.2.4.1.pom
The method CVEPut.create_pv_nodes needs to be covered by unit tests
The method CVEPut.create_pv_nodes from the script cve.py needs to be covered by unit tests
fabric8-analytics-data-model/src/cve.py
Line 58 in 7d93d83
Logging doesn't show messages from other than flask logger
There are many logger.<level>() other than from the REST API ( Flask ), which do not appear in the logs.
Gremlin queries are timing out in prod
Over 100 instances of Timing out messages in logs are encountered as shown below
Timing out script - g.V().has('ecosystem',ecosystem).has('name',within(input_packages)).as('pkg').out('has_version').hasNot('cve_ids').as('ver').select('pkg','ver').by(valueMap()).dedup() - in thread [gremlin-server-worker-1]
45814468 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor - Timing out script - g.V().has('ecosystem',ecosystem).has('name',within(input_packages)).as('pkg').out('has_version').hasNot('cve_ids').as('ver').select('pkg','ver').by(valueMap()).dedup() - in thread [gremlin-server-worker-1]
45815205 [gremlin-server-exec-4] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Error encountered evaluating script: g.V().has('ecosystem',ecosystem).has('name',within(input_packages)).as('pkg').out('has_version').hasNot('cve_ids').as('ver').select('pkg','ver').by(valueMap()).dedup()
org.apache.tinkerpop.gremlin.process.traversal.util.TraversalInterruptedException
at com.thinkaurelius.titan.diskstorage.BackendTransaction.executeRead(BackendTransaction.java:434)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.edgeStoreQuery(BackendTransaction.java:254)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.edgeQuery(StandardTitanGraph.java:413)
at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.lambda$get$0(SimpleVertexQueryProcessor.java:114)
at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:84)
at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:76)
at com.thinkaurelius.titan.graphdb.query.profile.QueryProfiler.profile(QueryProfiler.java:68)
at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.get(SimpleVertexQueryProcessor.java:114)
at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor$2.get(SimpleVertexQueryProcessor.java:111)
at com.thinkaurelius.titan.graphdb.vertices.CacheVertex.loadRelations(CacheVertex.java:53)
at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor.getBasicIterator(SimpleVertexQueryProcessor.java:111)
at com.thinkaurelius.titan.graphdb.query.vertex.SimpleVertexQueryProcessor.iterator(SimpleVertexQueryProcessor.java:68)
at com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.<init>(RelationConstructor.java:36)
at com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1.iterator(RelationConstructor.java:34)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.properties(AbstractVertex.java:160)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertyMapStep.map(PropertyMapStep.java:67)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.PropertyMapStep.map(PropertyMapStep.java:46)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.MapStep.processNextStart(MapStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:128)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.next(AbstractStep.java:38)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.next(DefaultTraversal.java:186)
at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalUtil.apply(TraversalUtil.java:93)
at org.apache.tinkerpop.gremlin.process.traversal.util.TraversalUtil.applyNullable(TraversalUtil.java:119)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.SelectStep.map(SelectStep.java:69)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.SelectStep.map(SelectStep.java:46)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.MapStep.processNextStart(MapStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.step.map.SelectStep.processNextStart(SelectStep.java:158)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:50)
at org.apache.tinkerpop.gremlin.process.traversal.step.filter.FilterStep.processNextStart(FilterStep.java:37)
at org.apache.tinkerpop.gremlin.process.traversal.step.filter.DedupGlobalStep.processNextStart(DedupGlobalStep.java:92)
at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:143)
at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:179)
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.fill(IteratorUtils.java:62)
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.list(IteratorUtils.java:85)
at org.apache.tinkerpop.gremlin.util.iterator.IteratorUtils.asList(IteratorUtils.java:382)
at org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler.lambda$channelRead$1(HttpGremlinEndpointHandler.java:239)
at org.apache.tinkerpop.gremlin.util.function.FunctionUtils.lambda$wrapFunction$0(FunctionUtils.java:36)
at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Some of the queries identified as timing out are:
g.V().has('ecosystem',ecosystem).has('name',within(input_packages)).as('pkg').out('has_version').hasNot('cve_ids').as('ver').select('pkg','ver').by(valueMap()).dedup() - in thread [gremlin-server-worker-1]
g.V().has('pecosystem', 'maven').has('pname', 'io.vertx:vertx-core').has('version', '3.4.1').as('version').in('has_version').as('package').select('version','package').by(valueMap()); - in thread [gremlin-server-worker-1]
g.V().has('pecosystem', 'maven').has('pname', 'io.vertx:vertx-web').has('version', '3.4.1').as('version').in('has_version').as('package').select('version','package').by(valueMap()); - in thread [gremlin-server-worker-1]
g.V().has('pecosystem', 'maven').has('pname', 'io.vertx:vertx-web-client').has('version', '3.4.1').as('version').in('has_version').as('package').select('version','package').by(valueMap()); - in thread [gremlin-server-worker-1]
manifest for registry.centos.org/datamodel/gremlin:concaf not found
When following local setup described in README.md, that is running
docker-compose -f docker-compose-server.yml upError message is produced:
ERROR: manifest for registry.centos.org/datamodel/gremlin:concaf not foundWhen going through https://registry.centos.org/containers/, there seems not to be such image.
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
[2018-02-22 17:49:32 +0000] [12] [ERROR] Error handling request /api/v1/sync_all?ecosystem=npm&package=08
--
| Traceback (most recent call last):
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
| self.handle_request(listener_name, req, client, addr)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
| super(GeventWorker, self).handle_request(*args)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
| respiter = self.wsgi(environ, resp.start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
| return self.wsgi_app(environ, start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
| response = self.handle_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
| response = self.full_dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
| rv = self.handle_user_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
| rv = self.dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
| return self.view_functions[rule.endpoint](**req.view_args)
| File "/src/rest_api.py", line 73, in sync_all
| version=version_id)
| File "/src/data_importer.py", line 252, in fetch_pending_epvs
| items = self.rdb.execute(query, params)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
| bind, close_with_result=True).execute(clause, params or {})
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
| engine, execution_options)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
| conn = bind.contextual_connect()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2106, in contextual_connect
| self._wrap_pool_connect(self.pool.connect, None),
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2145, in _wrap_pool_connect
| e, dialect, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception_noconnection
| exc_info
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
| reraise(type(exception), exception, tb=exc_tb, cause=cause)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2141, in _wrap_pool_connect
| return fn()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
| return _ConnectionFairy._checkout(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
| fairy = _ConnectionRecord.checkout(pool)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
| rec = pool._do_get()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
| self._dec_overflow()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
| compat.reraise(exc_type, exc_value, exc_tb)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
| return self._create_connection()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
| return _ConnectionRecord(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
| self.__connect(first_connect_check=True)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 661, in __connect
| exec_once(self.connection, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 246, in exec_once
| self(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
| fn(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 172, in on_connect
| do_on_connect(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 669, in on_connect
| fn(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 644, in on_connect
| hstore_oids = self._hstore_oids(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 775, in oneshot
| result = fn(self, *args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 678, in _hstore_oids
| oids = extras.HstoreAdapter.get_oids(conn)
| File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 917, in get_oids
| """ % typarray)
| DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
| server closed the connection unexpectedly
| This probably means the server terminated abnormally
| before or while processing the request.
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
As observed in logs from prod instance:
--------------------------------------------------------------------------------
--
| [2018-02-20 12:28:51 +0000] [268] [ERROR] Error handling request /api/v1/sync_all?ecosystem=maven&package=com.github.hotchemi%3Aandroid-rate
| Traceback (most recent call last):
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
| self.handle_request(listener_name, req, client, addr)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
| super(GeventWorker, self).handle_request(*args)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
| respiter = self.wsgi(environ, resp.start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
| return self.wsgi_app(environ, start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
| response = self.handle_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
| response = self.full_dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
| rv = self.handle_user_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
| rv = self.dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
| return self.view_functions[rule.endpoint](**req.view_args)
| File "/src/rest_api.py", line 73, in sync_all
| version=version_id)
| File "/src/data_importer.py", line 250, in fetch_pending_epvs
| items = self.rdb.execute(query, params)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
| bind, close_with_result=True).execute(clause, params or {})
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
| engine, execution_options)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
| conn = bind.contextual_connect()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2106, in contextual_connect
| self._wrap_pool_connect(self.pool.connect, None),
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2145, in _wrap_pool_connect
| e, dialect, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception_noconnection
| exc_info
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
| reraise(type(exception), exception, tb=exc_tb, cause=cause)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2141, in _wrap_pool_connect
| return fn()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
| return _ConnectionFairy._checkout(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
| fairy = _ConnectionRecord.checkout(pool)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
| rec = pool._do_get()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
| self._dec_overflow()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
| compat.reraise(exc_type, exc_value, exc_tb)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
| return self._create_connection()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
| return _ConnectionRecord(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
| self.__connect(first_connect_check=True)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 661, in __connect
| exec_once(self.connection, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 246, in exec_once
| self(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
| fn(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 172, in on_connect
| do_on_connect(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 669, in on_connect
| fn(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 644, in on_connect
| hstore_oids = self._hstore_oids(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 775, in oneshot
| result = fn(self, *args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 678, in _hstore_oids
| oids = extras.HstoreAdapter.get_oids(conn)
| File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 917, in get_oids
| """ % typarray)
| DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
| server closed the connection unexpectedly
| This probably means the server terminated abnormally
| before or while processing the request.
TODO: refactor into the separate module
Current graph queries could be missing some package versions
This is what I have noticed on production for few packages, for example org.wildfly.swarm:odata:
All analyzed versions for this package (using bookkeeping api):
"analysed_versions": [
"2012.12.0",
"2017.12.0",
"2017.12.1",
"2018.1.0",
"2018.2.0",
"2018.2.0.Final"
]
Which of these versions are not yet synced to graph ? ( using graph_sync pending API):
{
"data": [
{
"ecosystem": "maven",
"name": "org.wildfly.swarm:odata",
"version": "2017.12.0"
}
]
}
Even after invoking the graph sync API, I still get the pending version as not synced to graph.
{
"data": {
"count_imported_EPVs": 0,
"epv": [
{
"ecosystem": "maven",
"name": "org.wildfly.swarm:odata",
"version": "2017.12.0"
}
],
"message": "Nothing to be synced to Graph!"
}
}
Notice the message above "count_imported_EPVs": 0,. This clearly means that we have atleast one of three issues:
- Graph queries used to detect if a package is present in graph are incorrect
- Graph queries used to actually sync this package to graph are incorrect
- There is some other issue causing this behavior
Remove unused code
Current code coverage report is as below:
---------- coverage: platform linux2, python 2.7.14-final-0 ----------
Name Stmts Miss Cover Missing
-----------------------------------------------------------------------
src/__init__.py 2 0 100%
src/config.py 22 0 100%
src/data_importer.py 186 62 67% 69-70, 86, 97, 139, 158, 174, 189, 191-194, 213-215, 247-277, 294, 297-330, 333-356
src/data_source/__init__.py 0 0 100%
src/data_source/abstract_data_source.py 6 0 100%
src/data_source/s3_data_source.py 36 7 81% 29-31, 38, 43, 53, 63-64
src/graph_manager.py 38 13 66% 26-28, 35-39, 44-52, 57-68
src/graph_populator.py 178 24 87% 43, 61-63, 96-97, 116-117, 132, 142, 164, 167-169, 295-297, 307-309, 332-335, 352-360, 394
src/rest_api.py 97 65 33% 35-39, 45, 52, 58-71, 77-102, 108-126, 132-157, 161
src/sanitycheck.py 34 34 0% 3-59
src/set_logging.py 7 7 0% 3-12
-----------------------------------------------------------------------
TOTAL 606 212 65%
We don not use the last two files file:
src/set_logging.pysrc/sanitycheck.py
We can remove them.
TODO: refactor into the separate module
Error inserting last_updated property: java.math.BigDecimal is not supported
163856 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Property value [1499082539.8] is of type class java.math.BigDecimal is not supported
java.lang.IllegalArgumentException: Property value [1499082539.8] is of type class java.math.BigDecimal is not supported
at org.apache.tinkerpop.gremlin.structure.Property$Exceptions.dataTypeOfPropertyValueNotSupported(Property.java:163)
Remove gremlin-python imports from graph manager and requirements.txt
Check when no package-level data is present and others
Carried forward from another issue: openshiftio/openshift.io#2047 (comment)
package data (prod-bayesian-core-package-data) for a particular version is not present in S3 but package version does (prod-bayesian-core-data) have data in S3. For example there are 75 versions for arachne-ui but there is no package data present
I don't think that having no package-level data should be an issue for graph import. In case of arachne-ui, it looks like an error in ingestion pipeline, but in general, there can be components for which we simply won't have any package-level data.
Could you please check in data-importer, how it behaves when there are no package-level data? And file an issue, if it fails :) Thanks 😉
Adding more issues to investigate further:
openshiftio/openshift.io#2047 (comment)
The graph sync ran for few days now, and we have 92695 NPM package versions in the pending list. There are no more package-versions going through the sync anymore.
There are known reasons due to which this is happening
- package data (
prod-bayesian-core-package-data) for a particular version is not present in S3 but package version does (prod-bayesian-core-data) have data in S3. For example there are 75 versions forarachne-uibut there is no package data present. - there is no data at all in S3 for a particular package
- other reasons we haven't figured out yet.
Essentially for NPM, the graph-sync is done to the extent which it could be done in the current state.
ERROR:data_importer:The import failed: 'NoneType' object has no attribute 'split'
ERROR:data_importer:The import failed: 'NoneType' object has no attribute 'split'
--
| ERROR:data_importer:Traceback for latest failure in import call: Traceback (most recent call last):
| File "/src/data_importer.py", line 87, in _import_keys_from_s3_http
| str_gremlin = GraphPopulator.create_query_string(obj)
| File "/src/graph_populator.py", line 371, in create_query_string
| str_package, prp_package = cls.construct_package_query(input_json)
| File "/src/graph_populator.py", line 204, in construct_package_query
| latest_version = cls._sanitize_text_for_query(input_json.get('latest_version')) or ''
| File "/src/graph_populator.py", line 18, in _sanitize_text_for_query
| text = " ".join([l.strip() for l in text.split("\n")])
| AttributeError: 'NoneType' object has no attribute 'split'
|
| ERROR:data_importer:
| Report from import_epv():
| The import failed: 'NoneType' object has no attribute 'split'
| Total number of EPVs imported: 0
| The last successfully imported EPV: None
Convert the insertion functions to gremlin-http
This code branch in function get_qstring_for_cve_node needs to be covered by unit test
This code branch in function get_qstring_for_cve_node needs to be covered by unit test
fabric8-analytics-data-model/src/cve.py
Line 99 in 7d93d83
Data importer fails if main json missing in S3.
Reported by @geetikabatra
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
[2018-02-13 09:58:48 +0000] [13] [ERROR] Error handling request /api/v1/sync_all?package=org.webjars.npm%3Axml-writer&ecosystem=maven
| Traceback (most recent call last):
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
| self.handle_request(listener_name, req, client, addr)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
| super(GeventWorker, self).handle_request(*args)
| File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
| respiter = self.wsgi(environ, resp.start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
| return self.wsgi_app(environ, start_response)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
| response = self.handle_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
| response = self.full_dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
| rv = self.handle_user_exception(e)
| File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
| return cors_after_request(app.make_response(f(*args, **kwargs)))
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
| reraise(exc_type, exc_value, tb)
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
| rv = self.dispatch_request()
| File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
| return self.view_functions[rule.endpoint](**req.view_args)
| File "/src/rest_api.py", line 73, in sync_all
| version=version_id)
| File "/src/data_importer.py", line 250, in fetch_pending_epvs
| items = self.rdb.execute(query, params)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
| bind, close_with_result=True).execute(clause, params or {})
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
| engine, execution_options)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
| conn = bind.contextual_connect()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2106, in contextual_connect
| self._wrap_pool_connect(self.pool.connect, None),
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2145, in _wrap_pool_connect
| e, dialect, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception_noconnection
| exc_info
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
| reraise(type(exception), exception, tb=exc_tb, cause=cause)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2141, in _wrap_pool_connect
| return fn()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
| return _ConnectionFairy._checkout(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
| fairy = _ConnectionRecord.checkout(pool)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
| rec = pool._do_get()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
| self._dec_overflow()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
| compat.reraise(exc_type, exc_value, exc_tb)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
| return self._create_connection()
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
| return _ConnectionRecord(self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
| self.__connect(first_connect_check=True)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 661, in __connect
| exec_once(self.connection, self)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 246, in exec_once
| self(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
| fn(*args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 172, in on_connect
| do_on_connect(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 669, in on_connect
| fn(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 644, in on_connect
| hstore_oids = self._hstore_oids(conn)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 775, in oneshot
| result = fn(self, *args, **kw)
| File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 678, in _hstore_oids
| oids = extras.HstoreAdapter.get_oids(conn)
| File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 904, in get_oids
| """ % typarray)
| DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
| server closed the connection unexpectedly
| This probably means the server terminated abnormally
| before or while processing the request.
incorrect yaml specification in docker-compose-importer.yml
According to README.md a user should be running the following command to run data importer in a container:
sudo docker-compose -f docker-compose-importer.yml buildHowever, there seems to be incorrect yaml specification according to the error message produced:
yaml.scanner.ScannerError: while scanning a simple key
in "./docker-compose-importer.yml", line 27, column 7
could not find expected ':'
in "./docker-compose-importer.yml", line 28, column 7The .DS_Store file needs to be ignored in .gitignore file
The .DS_Store file needs to be ignored in .gitignore file
Strange error displayed to log during unit tests
See:
https://ci.centos.org/job/devtools-fabric8-analytics-data-model-fabric8-analytics/420/console
test/test_data_importer.py::test_get_exception_msg Top level Sentry exception caught - failed creating log record
b'prefix: hello world!'
b'Traceback (most recent call last):\n File "/root/payload/venv/lib/python3.4/site-packages/raven/handlers/logging.py", line 91, in emit\n self.format(record)\n File "/usr/lib64/python3.4/logging/__init__.py", line 828, in format\n return fmt.format(record)\n File "/usr/lib64/python3.4/logging/__init__.py", line 573, in format\n record.exc_text = self.formatException(record.exc_info)\n File "/usr/lib64/python3.4/logging/__init__.py", line 523, in formatException\n traceback.print_exception(ei[0], ei[1], tb, None, sio)\n File "/usr/lib64/python3.4/traceback.py", line 169, in print_exception\n for line in _format_exception_iter(etype, value, tb, limit, chain):\n File "/usr/lib64/python3.4/traceback.py", line 146, in _format_exception_iter\n for value, tb in values:\n File "/usr/lib64/python3.4/traceback.py", line 125, in _iter_chain\n context = exc.__context__\nAttributeError: \'NoneType\' object has no attribute \'__context__\'\n'
--- Logging error ---
Traceback (most recent call last):
File "/usr/lib64/python3.4/logging/__init__.py", line 978, in emit
msg = self.format(record)
File "/usr/lib64/python3.4/logging/__init__.py", line 828, in format
return fmt.format(record)
File "/usr/lib64/python3.4/logging/__init__.py", line 573, in format
record.exc_text = self.formatException(record.exc_info)
File "/usr/lib64/python3.4/logging/__init__.py", line 523, in formatException
traceback.print_exception(ei[0], ei[1], tb, None, sio)
File "/usr/lib64/python3.4/traceback.py", line 169, in print_exception
for line in _format_exception_iter(etype, value, tb, limit, chain):
File "/usr/lib64/python3.4/traceback.py", line 146, in _format_exception_iter
for value, tb in values:
File "/usr/lib64/python3.4/traceback.py", line 125, in _iter_chain
context = exc.__context__
AttributeError: 'NoneType' object has no attribute '__context__'
Call stack:
File "/root/payload/venv/bin/py.test", line 10, in <module>
sys.exit(main())
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/config/__init__.py", line 77, in main
return config.hook.pytest_cmdline_main(config=config)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 284, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/main.py", line 217, in pytest_cmdline_main
return wrap_session(config, _main)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/main.py", line 184, in wrap_session
session.exitstatus = doit(config, session) or 0
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/main.py", line 224, in _main
config.hook.pytest_runtestloop(session=session)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 284, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/main.py", line 245, in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 284, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 77, in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 92, in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 172, in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 196, in call_runtest_hook
treat_keyboard_interrupt_as_exception=item.config.getvalue("usepdb"),
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 212, in __init__
self.result = func()
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 194, in <lambda>
lambda: ihook(item=item, **kwds),
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 284, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/runner.py", line 122, in pytest_runtest_call
item.runtest()
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/python.py", line 1438, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/hooks.py", line 284, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 67, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/manager.py", line 61, in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/root/payload/venv/lib/python3.4/site-packages/pluggy/callers.py", line 187, in _multicall
res = hook_impl.function(*args)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call
testfunction(**testargs)
File "/root/payload/test/test_data_importer.py", line 68, in test_get_exception_msg
assert data_importer._get_exception_msg("prefix", e1) == "prefix: hello world!"
File "/root/payload/src/data_importer.py", line 55, in _get_exception_msg
tb = logging.exception(msg)
File "/usr/lib64/python3.4/logging/__init__.py", line 1814, in exception
error(msg, *args, **kwargs)
File "/usr/lib64/python3.4/logging/__init__.py", line 1805, in error
root.error(msg, *args, **kwargs)
File "/usr/lib64/python3.4/logging/__init__.py", line 1300, in error
Log 'msg % args' with severity 'ERROR'.
File "/usr/lib64/python3.4/logging/__init__.py", line 1308, in error
self._log(ERROR, msg, args, **kwargs)
File "/usr/lib64/python3.4/logging/__init__.py", line 1414, in _log
self.handle(record)
File "/usr/lib64/python3.4/logging/__init__.py", line 1424, in handle
self.callHandlers(record)
File "/usr/lib64/python3.4/logging/__init__.py", line 1486, in callHandlers
hdlr.handle(record)
File "/usr/lib64/python3.4/logging/__init__.py", line 853, in handle
self.emit(record)
File "/root/payload/venv/lib/python3.4/site-packages/_pytest/logging.py", line 206, in emit
logging.StreamHandler.emit(self, record)
Message: 'prefix: hello world!'
Arguments: ()
Top level Sentry exception caught - failed creating log record
b': hello world!'
b'Traceback (most recent call last):\n File "/root/payload/venv/lib/python3.4/site-packages/raven/handlers/logging.py", line 91, in emit\n self.format(record)\n File "/usr/lib64/python3.4/logging/__init__.py", line 828, in format\n return fmt.format(record)\n File "/usr/lib64/python3.4/logging/__init__.py", line 573, in format\n record.exc_text = self.formatException(record.exc_info)\n File "/usr/lib64/python3.4/logging/__init__.py", line 523, in formatException\n traceback.print_exception(ei[0], ei[1], tb, None, sio)\n File "/usr/lib64/python3.4/traceback.py", line 169, in print_exception\n for line in _format_exception_iter(etype, value, tb, limit, chain):\n File "/usr/lib64/python3.4/traceback.py", line 146, in _format_exception_iter\n for value, tb in values:\n File "/usr/lib64/python3.4/traceback.py", line 125, in _iter_chain\n context = exc.__context__\nAttributeError: \'NoneType\' object has no attribute \'__context__\'\n'
Query breaks due to quotes - version may also contain quotes
As per log from instance running on production:
85125493 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
--
| Script5383.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
|
| 1 error
|
| org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
| Script5383.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
The method CVEDelete.process needs to be covered by unit tests
The method CVEDelete.process from the script cve.py needs to be covered by unit tests:
fabric8-analytics-data-model/src/cve.py
Line 132 in 7d93d83
Cyclomatic complexity is too high for two methods in this repository
Please take a look and try to reduce cyclomatix complexity for the following functions:
src/graph_populator.py
M 106:4 GraphPopulator.construct_version_query - E (34)
M 272:4 GraphPopulator.construct_package_query - D (27)
2 blocks (classes, functions, methods) analyzed.
Average complexity: E (30.5)
Internal Server Error and startup failed: version contains quotes
1601524 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
--
| Script225.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
|
| 1 error
|
| org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
| Script225.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
|
| 1 error
|
| at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
| at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
| at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:145)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
| at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
| at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
| at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
| at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
| at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
| at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
| at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
| at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
| at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
| 1603188 [gremlin-server-worker-1] WARN org.apache.tinkerpop.gremlin.server.handler.HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and startup failed:
| Script226.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
|
| 1 error
|
| org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
| Script226.groovy: 1: expecting ')', found '0.11' @ line 1, column 1088.
| al.jdk:core').has('version',''0.11.3.0''
| ^
|
| 1 error
|
| at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
| at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:150)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:120)
| at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:132)
| at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:360)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:145)
| at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:111)
| at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:237)
| at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:167)
| at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:931)
| at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:593)
| at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:569)
| at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:546)
| at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
| at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:211)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:527)
| at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:446)
| at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
| at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:119)
| at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:287)
| at java.util.concurrent.FutureTask.run(FutureTask.java:266)
| at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
| at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
| at java.lang.Thread.run(Thread.java:748)
TODO: refactor into the separate module
Add support for bounded graph queries
At present BayesianGraph.execute() does not support queries with bounded parameters:
fabric8-analytics-data-model/src/graph_manager.py
Lines 16 to 30 in 13bd745
Support for bounded parameters will help us write better queries and avoid many bugs.
Failing test_insertion()
See for example https://ci.centos.org/job/devtools-fabric8-analytics-data-model-fabric8-analytics/14/consoleFull
The tests "pass", because the failure is silenced by try: except:, but it's there:
ERROR:test_insertion_from_minio:Traceback for latest failure in import call: Traceback (most recent call last):
File "/root/payload/test/test_insertion_from_minio.py", line 85, in test_insertion
assert report["epv"] == ["pypi:access_points:0.4.59"]
AssertionError: assert [{'pypi/acces...json'], ...}}] == ['pypi:access_points:0.4.59']
At index 0 diff: {'pypi/access_points/': {'version': '0.4.59', 'package': 'access_points', 'ecosystem': 'pypi', 'pkg_list_keys': ['pypi/access_points/github_details.json', 'pypi/access_points/libraries_io.json'], 'ver_key_prefix': 'pypi/access_points/0.4.59', 'pkg_key_prefix': 'pypi/access_points/', 'ver_list_keys': ['pypi/access_points/0.4.59/code_metrics.json', 'pypi/access_points/0.4.59/metadata.json', 'pypi/access_points/0.4.59/security_issues.json', 'pypi/access_points/0.4.59/source_licenses.json']}} != 'pypi:access_points:0.4.59'Check Python version in all CI jobs
Check if Python version installed on CI slaves are at least 3.6 or newer.
Route /api/v1/import_from_s3 is not working.
S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7fa6e6346950>>
DEBUG:botocore.hooks:Event after-call.s3.ListObjects: calling handler <function decode_list_object at 0x7fa6ea78b410>
DEBUG:boto3.resources.factory:Loading s3:ObjectSummary
TODO: reduce cyclomatic complexity
Update this repository to use Python 3.6 instead of Python 3.4
EPEL repositories now contain proper Python 3.6 packages and at the same moment Python 3.4 is being deprecated [1] [2].
It means that we need to upgrade this repository to use Python 3.6 instead of Python 3.4.
What needs to be changed AND tested:
- all Dockerfiles
- CICO setup
- linter and pydocstyle scripts
- CI and MI measurement scripts
- script to start tests
References:
[1] https://lists.fedoraproject.org/archives/list/[email protected]/thread/EGUMKAIMPK2UD5VSHXM53BH2MBDGDWMO/
[2] https://www.reddit.com/r/CentOS/comments/azetyy/python_34_to_be_deprecated_this_month/
The method CVEPut.process needs to be covered by unit tests
The method CVEPut.process from the script cve.py needs to be covered by unit tests:
fabric8-analytics-data-model/src/cve.py
Line 46 in 7d93d83
Python 2-related code in the rest_api.py
Do we plan to remove all the Python 2-related code from this repository?
api_v1 = Blueprint('api_v1', __name__)
# Python2.x: Make default encoding as UTF-8
if sys.version_info.major == 2:
reload(sys)
sys.setdefaultencoding('UTF8')
IMHO it is not much relevant these days...
Unit test failure
=================================== FAILURES ===================================
__________________________ test_ingest_to_graph_valid __________________________
client = <FlaskClient <Flask 'data_importer_app'>>
def test_ingest_to_graph_valid(client):
"""Add test for ingest to graph API when some key is missing."""
input_data = [
{
"ecosystem": "maven",
"name": "commons-collections:commons-collections",
"source_repo": "redhat_maven"
}
]
url = url_for('api_v1.ingest_to_graph')
response = client.post(url,
data=json.dumps(input_data),
headers={'Content-Type': 'application/json'})
# we expect that the HTTP code will be 400/Bad Request
assert response.status_code == 400
data = json.loads(response.get_data().decode('utf-8'))
epv_keys = input_data[0].keys()
> assert data['message'] == 'Invalid keys found in input: ' + ','.join(epv_keys)
E AssertionError: assert 'Invalid keys...m,source_repo' == 'Invalid keys ...cosystem,name'
E - Invalid keys found in input: name,ecosystem,source_repo
E ? ^^^ ^^^^^ -----
E + Invalid keys found in input: source_repo,ecosystem,name
E ? ^^^^^ +++++ ^^^
test/test_rest_api.py:122: AssertionError
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
Database fails to respond for some queries as shown in logs below. This however is not recurring, so adding here for tracking if it happens again:
--------------------------------------------------------------------------------
INFO in rest_api [/src/rest_api.py:28]:
Ready to serve requests
--------------------------------------------------------------------------------
[2018-02-06 06:04:44 +0000] [44] [ERROR] Error handling request /api/v1/sync_all?package=javax.xml%3Ajaxrpc-api-osgi&ecosystem=maven
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
super(GeventWorker, self).handle_request(*args)
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/src/rest_api.py", line 73, in sync_all
version=version_id)
File "/src/data_importer.py", line 250, in fetch_pending_epvs
items = self.rdb.execute(query, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
bind, close_with_result=True).execute(clause, params or {})
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
engine, execution_options)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
conn = bind.contextual_connect()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2106, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2145, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2141, in _wrap_pool_connect
return fn()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
rec = pool._do_get()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
self._dec_overflow()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
return self._create_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
self.__connect(first_connect_check=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 661, in __connect
exec_once(self.connection, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 246, in exec_once
self(*args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
fn(*args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 172, in on_connect
do_on_connect(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 669, in on_connect
fn(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 644, in on_connect
hstore_oids = self._hstore_oids(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 775, in oneshot
result = fn(self, *args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 678, in _hstore_oids
oids = extras.HstoreAdapter.get_oids(conn)
File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 904, in get_oids
""" % typarray)
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
--------------------------------------------------------------------------------
INFO in rest_api [/src/rest_api.py:128]:
Selective Ingestion with payload - {"select_ingest": ["keywords_tagging", "git_stats", "libraries_io", "github_details"], "package_list": [{"ecosystem": "maven", "version": null, "name": "com.lihaoyi:mill-moduledefs_2.12"}]}
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
INFO in rest_api [/src/rest_api.py:136]:
{'epv': {u'select_ingest': [u'keywords_tagging', u'git_stats', u'libraries_io', u'github_details'], u'package_list': [{u'ecosystem': u'maven', u'version': '', u'name': u'com.lihaoyi:mill-moduledefs_2.12'}]}, 'message': 'Nothing to be synced to Graph!', 'count_imported_EPVs': 0}
--------------------------------------------------------------------------------
[2018-02-06 06:17:59 +0000] [44] [ERROR] Error handling request /api/v1/sync_all?ecosystem=maven&package=net.artsy%3Aatomic-store_2.11
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 56, in handle
self.handle_request(listener_name, req, client, addr)
File "/usr/lib/python2.7/site-packages/gunicorn/workers/ggevent.py", line 152, in handle_request
super(GeventWorker, self).handle_request(*args)
File "/usr/lib/python2.7/site-packages/gunicorn/workers/async.py", line 107, in handle_request
respiter = self.wsgi(environ, resp.start_response)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib64/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/src/rest_api.py", line 73, in sync_all
version=version_id)
File "/src/data_importer.py", line 250, in fetch_pending_epvs
items = self.rdb.execute(query, params)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1107, in execute
bind, close_with_result=True).execute(clause, params or {})
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 971, in _connection_for_bind
engine, execution_options)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 403, in _connection_for_bind
conn = bind.contextual_connect()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2106, in contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2145, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2141, in _wrap_pool_connect
return fn()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 387, in connect
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
rec = pool._do_get()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
self._dec_overflow()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
return self._create_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
self.__connect(first_connect_check=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 661, in __connect
exec_once(self.connection, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 246, in exec_once
self(*args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
fn(*args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 172, in on_connect
do_on_connect(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 669, in on_connect
fn(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 644, in on_connect
hstore_oids = self._hstore_oids(conn)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 775, in oneshot
result = fn(self, *args, **kw)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 678, in _hstore_oids
oids = extras.HstoreAdapter.get_oids(conn)
File "/usr/lib64/python2.7/site-packages/psycopg2/extras.py", line 904, in get_oids
""" % typarray)
DatabaseError: (psycopg2.DatabaseError) query_wait_timeout
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
MultipleCompilationErrorsException: startup failed: unexpected char: '\'
Following errors happen due to unexpected characters in the query strings:
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script782.groovy: 1: unexpected char: '\' @ line 1, column 7263.
mber value using Number.equals\ boolean
^
--
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script783.groovy: 1: unexpected char: '\' @ line 1, column 7284.
mber value using Number.equals\ boolean
^
--
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script1123.groovy: 1: unexpected char: '\' @ line 1, column 11138.
\'/\' withBody name/\w/ \'harr
^
--
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script1125.groovy: 1: unexpected char: '\' @ line 1, column 11138.
\'/\' withBody name/\w/ \'harr
^
1 error
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script1149.groovy: 1: unexpected char: '\' @ line 1, column 2867.
hodquotPOSTquot .bodyquot\quotname\
^
Following file contains full error logs:
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.