Is it possible to speed up initial import? about trade-dangerous HOT 3 CLOSED

The script only works on listings.csv, whereas the plugin also needs to import listings-live.csv

The script doesn't check to make sure that the listings it's checking to import is newer than the listing it would replace, it just inserts or replaces everything. If the data in the DB is newer than the data from the import file, it should be skipping that item, and it isn't.

What should be done, and hasn't mainly due to lack of time, is to not perform each transaction individually, which is what is currently happening, but to push all the transactions into a list for an executemany call.

The biggest slowdown with the current method is that each transaction is done individually, so the DB gets opened, written, closed, open, written, closed, for every entry in the import file.

It's the constant opening and closing that is the biggest slowdown.

I originally wrote the plugin with literally 0 knowledge of SQLite3 beyond what I was Googling to get the thing to work, I just haven't gone back to redo the importing in a less stupid way.

TBH, every one of the import methods in the plugin really needs to be improved in the same manner.

from trade-dangerous.

Over my head, but there are some examples of executemany in cache.py if that helps.

from trade-dangerous.

This should probably be closed now due to 48686db, no?

from trade-dangerous.