Comments (36)
It seems that you have no indexes in the whoosh directory. It could be that the indexing was not done at the initial import. You can run manually the indexing by doing the following command:
python3 db_fulltext.py -l 0 -v
and then do search again using search_fulltext.py. If you any issue, let me know.
from cve-search.
@adulau Thanks for reply.
I executed the command python3 db_fulltext.py -l 0 -v
and after a few minutes, ended correctly.
But running the command python3 search_fulltext.py -g -s >cve.json
Same error:
XXXX@ubuntu:~/cve-search/bin$ python3 search_fulltext.py -g -s >cve.json
Traceback (most recent call last):
File "search_fulltext.py", line 28, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
Thanks
from cve-search.
hi @adulau
Any news?
Thanks :D
from cve-search.
It seems that the indexdir is not at the right place. Could you check where you Whoosh indexes were created? You can update it from the tool itself (I still need to update the code to use the directory path from the config file)
from cve-search.
@adulau Thanks Alexandre.
But.. how check where my Whoosh indexdir were created?
I am very very newbie in linux.
I googled it (whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')) but i couldn't find any solution
@PidgeyL This error occurs in the same configuration bug that you helped me fix it.
You can check if you have my new problem ?
from cve-search.
Hi @adulau @PidgeyL
Any News?
Thanks! :D
from cve-search.
Hello @adulau @PidgeyL
Any News?
I tried searching on Google but not found anything.
Thanks!
from cve-search.
Could you just check if you have the index built by db_fulltext.py? it's in indexdir with files likes MAIN_*.
from cve-search.
@adulau Thanks for reply.
I executed the command python3 db_fulltext.py -l 0 -v
and after a few minutes, ended correctly.
But running the command python3 search_fulltext.py -g -s >cve.json
Same error :(
I have a folder called " idexdir " with these files
This is correct? Or wrong?
from cve-search.
I didn't search for related issues when I opened #115, sorry for that.
Had some time to do some additional debugging today:
If I change indexpath = "./indexdir"
to indexpath = "/home/me/Source/cve-search/indexdir/"
(the directory of the index) in search_fulltext.py it seems to work:
me@zuuuul:~/Source/cve-search/bin$ python3 search_fulltext.py
usage: search_fulltext.py [-h] [-q Q] [-t] [-f] [-m M] [-l] [-g] [-s] [-n]
[-r]
Full text search for cve-search
optional arguments:
-h, --help show this help message and exit
-q Q query to lookup (one or more)
-t output title of the match CVE(s)
-f output matching CVE(s) in JSON
-m M most frequent terms in CVE description (m is top-m values)
-l dump all terms encountered in CVE description
-g graph of most frequent terms with each matching CVE (JSON
output)
-s enable stemming on graph JSON output (default is False)
-n lookup complete cpe (Common Platform Enumeration) name for
vulnerable configuration
-r lookup ranking of vulnerable configuration
me@zuuuul:~/Source/cve-search/bin$
I'll keep looking as well.
from cve-search.
thanks @tunkaflux
I tried what you say and do not work for me :(
in the folder /home/XXX/cve-search/bin
I modified the file search_fulltext.py
for:
indexpath = "/home/XXX/cve-search/sbin/indexdir/"
Then, i run
/home/XXX/cve-search/sbin/python3 db_fulltext.py
And Same error.
XXX@XXX:~/cve-search/bin$ python3 search_fulltext.py -g -s >cve.json
Traceback (most recent call last):
File "search_fulltext.py", line 28, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
XXX@XXX:~/cve-search/bin$
this is strange , because in the folder indexdir files MAIN exist
XXX@XXX:~/cve-search/sbin/indexdir$ ls
_MAIN_182025.toc MAIN_mrk9sxg7a3yov83a.seg MAIN_t0njwzx0d0erltv3.seg
MAIN_2sqhgn9dgsziop36.seg MAIN_n5k6rxn2bhlno8c3.seg MAIN_txtxc3b7k3ehqa6f.seg
MAIN_87ifw0zj2zi56f13.seg MAIN_ophh6mwh8wiiejlz.seg MAIN_WRITELOCK
MAIN_gh9o0059dz16kmzh.seg MAIN_pbf1ag3eusak09bb.seg MAIN_zczkznex6ipiiwnt.seg
MAIN_i89a142xfe6be5vk.seg MAIN_qpxm9zfq73e0xpbw.seg
MAIN_lz4f7r48zqz662p3.seg MAIN_r7e3w0au0zfiia19.seg
from cve-search.
Ok... but why did you close this? :)
from cve-search.
@tunkaflux Sorry, my bad :P
from cve-search.
Also, I noticed my indexdir is in "/home/xxx/cve-search/indexdir/". That's also the directory that gets updated when I reindex everything.
It seems you are using "/home/xxx/cve-search/sbin/indexdir"? Can you reindex everyhing and see which dir gets updated: cve-search/indexdir or cve-search/sbin/indexdir?
Afterwards, point to the updated indexdir and try again.
from cve-search.
For the record, both indexdirs work for me, so are you sure that you didn't make a typo somewhere?
from cve-search.
@tunkaflux
Thank you very much for answering
This is strange, i reindex everything and still creates the MAIN files in /sbin/indexdir
but search_fulltext.py
not working.
I re- download CVE-SERCH and I used the command ( who told me @adulau) python3 db_fulltext.py -l 0 -v
to reindex and not working.
Also modified the file search_fulltext.py
(change indexpath) and I still generates the same problem
Which command you used to reindex everything?
What operating system and version used? So I try the same specifications that you have
Thanks!
from cve-search.
I'm running Ubuntu 14.04, 64 and 32 bit
from cve-search.
Ubuntu 14.04, 64 bit here.
from cve-search.
Hi!
Not work for me :( Install a new Ubuntu (14.04) and install/extract CVE-SEARCH (new dowload) and the same error.
python3 search_fulltext.py
Traceback (most recent call last):
File "search_fulltext.py", line 30, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
BTW, now the MAIN files are in the folder "cve-search/indexdir"
any ideas? @adulau @tunkaflux @PidgeyL
If you want, I can upload my image VirtualBox Ubuntu so they can check or revise my problem.
from cve-search.
Hi people!
any news or suggestions?
from cve-search.
@sugusrojo Have you tried fix #97 from the Adulau repo? See commit (adulau@98bda7d)
from cve-search.
@tunkaflux tThanks for the Reply!
Yes, I've tried.
I tried new fresh installation Ubuntu 14.04 and re dowloding a new copy CVE-SEARCH.
now, the MAIN files are in the folder "cve-search/indexdir" but the same error occurs.
from cve-search.
@tunkaflux you can upload your Ubuntu Virtual Image?
Because...
You CVE-SEARCH in Ubuntu works.
My CVE-SEARCH not work in Ubuntu .
I want to see the difference between these two OS configuration or see what's difference.
from cve-search.
@sugusrojo No, unfortunately, I can't upload an image.
@adulau @PidgeyL: Does search work for you guys?
from cve-search.
I've tried it as well and got the same kind of error. I let the VM run at the office as it's currently generating this index. I'll retry tomorrow morning when it's done.
PS: Running this on Ubuntu Server 14.04-3 LTS x86_64.
from cve-search.
Thanks @AkiraTakizawa . I'll wait for your answer.
from cve-search.
The last commit should be fine.
Here is an example of the use of the full-text indexing:
[adulau:~/git/cve-search]↥ master* ± python3 ./sbin/db_fulltext.py -l 10 -v
Indexing CVE-ID CVE-2015-8581 The EjbObjectInputStream class in Apache TomEE allows remote attackers
Indexing CVE-ID CVE-2015-8580 Multiple use-after-free vulnerabilities in the (1) Print method and (2
Indexing CVE-ID CVE-2015-8566 The Session package 1.x before 1.3.1 for Joomla! Framework allows remo
Indexing CVE-ID CVE-2015-8565 Directory traversal vulnerability in Joomla! 3.2.0 through 3.3.x and 3
Indexing CVE-ID CVE-2015-8564 Directory traversal vulnerability in Joomla! 3.4.x before 3.4.6 allows
Indexing CVE-ID CVE-2015-8563 Cross-site request forgery (CSRF) vulnerability in the com_templates c
Indexing CVE-ID CVE-2015-8562 Joomla! 1.5.x, 2.x, and 3.x before 3.4.6 allow remote attackers to con
Indexing CVE-ID CVE-2015-8476 Multiple CRLF injection vulnerabilities in PHPMailer before 5.2.14 all
Indexing CVE-ID CVE-2015-8370 Multiple integer underflows in Grub2 1.98 through 2.02 allow physicall
Indexing CVE-ID CVE-2015-8358 Directory traversal vulnerability in the bitrix.mpbuilder module befor
Processing[##################################################] 10/10
[adulau:~/git/cve-search]↥ master* ± python3 ./bin/search_fulltext.py -q TomEE
CVE-2015-8581
[adulau:~/git/cve-search]↥ master* ±
I hope this helps.
from cve-search.
Good morning
I confirm that it works after (re)building the index :)
Example:
sadao@kujira:~/cve-search$ python3 ./bin/search_fulltext.py -q Lync
CVE-2013-1302
CVE-2015-2431
CVE-2014-4068
CVE-2015-2510
CVE-2015-6061
CVE-2014-4071
CVE-2015-1671
CVE-2015-2532
CVE-2015-2435
CVE-2014-4070
CVE-2015-2536
CVE-2014-1823
CVE-2012-1849
CVE-2015-2464
CVE-2015-2463
CVE-2015-2531
CVE-2015-2456
CVE-2015-2455
CVE-2014-1818
CVE-2014-1817
CVE-2015-6106
CVE-2015-6107
CVE-2015-6108
CVE-2012-1858
CVE-2013-3906
CVE-2012-2520
CVE-2013-3129
CVE-2015-2503
from cve-search.
@AkiraTakizawa Thank you for the test.
from cve-search.
I found the problem! @adulau @AkiraTakizawa @tunkaflux @PidgeyL
If I run the command from the folder CVE-search
WORKS
XXX@XXX:~/cve-search$ python3 ./bin/search_fulltext.py -q CVE-2013-1455
If I run it from the CVE-search\bin
folder does NOT WORK! It fails.
XX@CXXX:~/cve-search/bin$ python3 search_fulltext.py -q CVE-2013-1455
Traceback (most recent call last):
File "search_fulltext.py", line 30, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
Example:
from cve-search.
Thanks for your help! That probably means there is still a relative path that is not working yet. I'll take this issue and work on it on my way home from work, and if fixed, I'll commit it tonight
from cve-search.
I am currently testing this, but it works perfectly for me in both cases. Can you do the following for me?
- give me the value for Indexdir in el./etc/configuration.ini
- add print(indexpath) to line 27 of ./bin/search_fulltext.py and give the output?
from cve-search.
Thanks @PidgeyL
- I haven't file configuration.ini in cve-search/etc.
Only "configuration.ini.sample"
[Redis]
Host: localhost
Port: 6379
VendorsDB: 10
NotificationsDB: 11
RefDB: 12
[Mongo]
Host: localhost
Port: 27017
DB: cvedb
[dbmgt]
Tmpdir: ./tmp/
[FulltextIndex]
Indexdir: ./indexdir/
[Sources]
CVE: http://static.nvd.nist.gov/feeds/xml/cve/
CPE: http://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.2.xml
CWE: http://cwe.mitre.org/data/xml/cwec_v2.8.xml.zip
d2sec: http://www.d2sec.com/exploits/elliot.xml
vFeed: http://www.toolswatch.org/vfeed/vfeed.db.tgz
vFeedStatus: http://www.toolswatch.org/update.dat
Vendor: https://nvd.nist.gov/download/vendorstatements.xml
CAPEC: http://capec.mitre.org/data/xml/capec_v2.6.xml
MSBULLETIN: http://download.microsoft.com/download/6/7/3/673E4349-1CA5-40B9-8879-095C72D5B49D/BulletinSearch.xlsx
Ref: https://cve.mitre.org/data/refs/refmap/allrefmaps.zip
exploitdb: https://github.com/offensive-security/exploit-database/raw/master/files.csv
[Webserver]
Host: 127.0.0.1
Port: 5000
Debug: True
PageLength: 50
LoginRequired: False
SSL: True
Certificate: ssl/cve-search.crt
Key: ssl/cve-search.key
[Logging]
Logging: True
Logfile: log/cve-search.log
MaxSize: 150MB
Backlog: 5
[CVE]
DefaultCVSS: 5
StartYear: 2002
Add line 27
/home/XXX/cve-search/lib/.././indexdir
Traceback (most recent call last):
File "search_fulltext.py", line 30, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
from cve-search.
And if you do step two from the above directory?
from cve-search.
It's strange @PidgeyL
Both in the same location!
But fails from CVE-search \ bin
Thanks
csirt@CSIRTCVE:~/cve-search$ python3 ./bin/search_fulltext.py -q CVE-2013-145
/home/csirt/cve-search/lib/.././indexdir
csirt@CSIRTCVE:~/cve-search/bin$
csirt@CSIRTCVE:~/cve-search$ cd bin
csirt@CSIRTCVE:~/cve-search/bin$ python3 search_fulltext.py -q CVE-2013-145
/home/csirt/cve-search/lib/.././indexdir
Traceback (most recent call last):
File "search_fulltext.py", line 30, in <module>
ix = index.open_dir("indexdir")
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 123, in open_dir
return FileIndex(storage, schema=schema, indexname=indexname)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 421, in __init__
TOC.read(self.storage, self.indexname, schema=self._schema)
File "/usr/local/lib/python3.4/dist-packages/whoosh/index.py", line 619, in read
% (indexname, storage))
whoosh.index.EmptyIndexError: Index 'MAIN' does not exist in FileStorage('indexdir')
csirt@CSIRTCVE:~/cve-search/bin$
from cve-search.
I'm baffled... Could you gzip your entire folder and send it to me, either by mail or upload it somewhere? That would be a lot easier to debug
from cve-search.
Related Issues (20)
- Download issues with db_mgmt_json.py HOT 5
- ModuleNotFoundError: No module named 'sqlalchemy' - db_updater.py -v HOT 3
- Tailored the New Vulnerabilities CVE only with list of Assets HOT 1
- CPE population fails with urllib3 1.x (stable version on Ubuntu 22.04 LTS) HOT 4
- New releases? HOT 16
- Update MongoDB in documentation HOT 1
- CPE parsing problem
- CVE-Search lifespan / roadmap HOT 2
- add plugins (steps) HOT 1
- New bug when Populating the database HOT 3
- Running as a service (version 5.0.2 HOT 2
- cve-search database doesn't populate even after forcing and dropping everything HOT 2
- Proxy setting not passed to CveXplore
- CVE and CPE Imports and Updates fail HOT 2
- Check for missed CVE's HOT 5
- Unable to create user accounts with db_mgmt_admin.py HOT 9
- CVE search for "curl:curl:8.6.0" does not return CVE-2024-2004 HOT 1
- ERROR: Could not find a version that satisfies the requirement cvexplore==0.3.30 HOT 1
- AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'. Did you mean: 'X509_V_FLAG_EXPLICIT_POLICY'? HOT 12
- Unable to connect to mongodb with user/password
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cve-search.