valdergallo / data-importer Goto Github PK

set_reader need clean memory and update _readed to False.
To clean_data reread data file and validate

I'm Getting an ImoprtError "cannot import name UnsuportedFile"

It's on import line "from data_importer.core.exceptions import UnsuportedFile" on file generic.py

Need remove error line from clenaed_data

Add https://github.com/whitehat2k9/PyExcelerate

Benchmarks

Benchmark code located in pyexcelerate/tests/benchmark.py
65000 rows x 100 columns of the number 1
Ubuntu 12.04 LTS, Core i3-2310M 2.1GHz, 8GB DDR3, Python 2.7.3

PyExcelerate 74.29s
openpyxl: 757.29s
xlsxwriter: 120.12s

Use descriptor file to importer like Meta.model.

Ex.:

import_test.json

{
  'app_name': 'mytest.Contact',
    { 
    // field name / name on import file or key index
    'name': 'My Name',
    'year': 'My Year',
    'last': 3
    }
}

model.py

class Contact(models.Model):
  name = models.CharField(max_length=50)
  year = models.CharField(max_length=10)
  laster = models.CharField(max_length=5)

importer.py

class MyImpoter(BaseImpoter):
  class Meta:
    config_file = 'import_test.json'
    model = Contact
    delimiter = ','
    ignore_first_line = True

content_file.csv

name,year,last
Test,12,1
Test2,13,2
Test3,14,3

Create get_file_header, and udpate data with file header.

Data-impoter must be avaliable to all Djangos

Create one pre_validate for all lines in options.

The only folder currently included is the importers folder.

Add in documentation link to github issues for report bugs and issues

Default settings with customize options

Create one default TemplateView that save one FileHistory and run Importer

Wenze van Klink

I have a problem with installing your app.

When using Pip:
IOError: [Errno 2] No such file or directory: '/tmp/pip-build-wenze/data-importer/README.rst'

And when I install it manually I cannot import it in my IDE (PyCharm), but all the requirements are installed correctly:
ImportError: No module named data_importer

Hope you can help me out.

Manager customer messages with Markdown by URL

Create option to define the line that import must start to import

Add settings configuration to delete file from system on delete register

Some fields have Field validator, we can use this validator to validate data import

Data-importer don't need Celery. But if you has instaled Celery or need use importer as Task. Default FileUploadHistoryForm must have one option to use Data-importer as Task.

cleaned_data is one tuple with all file data, in large files, cleaned_data will use alot memory.
Need create one leazy object to cleaned_data that don't use memory

File base.py has a bug. This says the console:

except StopImporter, e: in line 198.

Need create one cleaned_line, that contains only data in row to validate.

cleaned_data contains all lines that was cleaned.

Change charset to UTF-8 to defult value in settings used by django