how to read faster about dbfread HOT 4 CLOSED

I've done all I could think of to make dbfread fast but unfortunately it has to do a lot of processing for each record.

One thing you could try is to use dictionaries instead of OrderedDict:

for record in DBF(filepath, recfactory=dict):
    ...

dbfread uses OrderedDict as default for convenience, but this adds some overhead.

Passing recfactory=None will give you (name, value) tuples which should be even faster but probably harder to work with.

Depending on your data you could read the file in raw mode and decode the byte strings yourself. This would save a lot of decoding if you're only using some of the fields:

for record in DBF(filepath, raw=True):
    name = record['name'].decode('latin1')
    age = int(record['age'])

Passing lowernames=False could help a tiny bit as it would save a call to .lower().

I'll see if I can think of anything else.

from dbfread.

Thanks for the advices!

from dbfread.

Hi, I have a mistake, most my time is consumed in the simple code, not in read dbf file.
I have changed the code inside for loop, and it speed up.
Sorry for the mistake and thank you very much for this module!

This issue is no longer a issue for me.
You can close this issue if you do not plan to speed up the reading process. :-P

from dbfread.

Ah, that's good news. I'll close the issue.

from dbfread.