azavea / django-queryset-csv Goto Github PK
View Code? Open in Web Editor NEWa CSV exporter for django querysets
License: Other
a CSV exporter for django querysets
License: Other
Leaning too heavily on StringIO
to mock a csv file. There needs to be an easy way to produce CSVs from this repo, so that they can be poked at in various CSV editors. This can either be done in the unit tests, or with views in the test_app
.
Hi, I export a file containing ñáéíóú characters pretty common in other languages, but it's rendering Ñ instead of ñ, as well as other encoding for the accents.
I don't know what is generating the issue, I saw you gave csv_kwargs = {'encoding': 'utf-8'}
pip freeze gave me django-queryset-csv==1.0.1
django-queryset-csv/djqscsv/djqscsv.py
Line 83 in 9eccc41
There's a bit if code duplication in the unit tests. It's a little hard to reason about what's under test. 100% test coverage is great, but quality is more important.
Hii, there I believe this is a really nicely done work and it is going to help in my work but don't you think we can also have a reverse of this which imports the data from CSV into the database. If you have some thoughts on it let me know, I am more than happy to contribute.
It would be great if the library would support writing lists
to csv.
Something like this:
header = [...]
data = [[], [], []]render_to_csv_response(data, header=header)
Models:
class Model():
field1...
field2...
...
class ModelB():
fk(Model)...
field3....
field4....
...
class ModelC():
fk(Model)
field5
field6
...
For example:
data = Model.objects.all()
for obj in data:
ModelReverse = obj.modelb_set.all()
OtherModel = obj.modelc_set.all()
etc .....
is posible output in only single file???
similar this:
field1,field2,field3,field4,field5,field6
https://github.com/azavea/django-queryset-csv/blob/master/djqscsv/__init__.py#L74 should be:
response['Content-Disposition'] = 'attachment; filename="%s";' % filename_param
export csv has  on the first column, looks like a pretty common issue
It only happened inside the first colomn description
I want you to take title and get Título
Anexo.csv:
ID,Título,fichero,orden
1,Maniobras de reanimación. Soporte vital básico con y sin desfibrilador,anexos/a1.pdf,0
views.py:
class Anexo(models.Model):
# Anexo Fields
title = models.CharField(_(u'Título'), max_length=255)
fichero = models.FileField(upload_to="anexos")
orden = models.PositiveSmallIntegerField(default=0, blank=False, null=False)
def __unicode__(self):
return strip_tags(self.title)
Downloading/unpacking django-queryset-csv>=0.2.4 (from -r ../../env/requirements.txt
(line 25))
Running setup.py egg_info for package django-queryset-csv
Traceback (most recent call last):
File "", line 16, in
File "/usr/local/otm/env/build/django-queryset-csv/setup.py", line 10, in
long_description=open('README.rst').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 16, in
File "/usr/local/otm/env/build/django-queryset-csv/setup.py", line 10, in
long_description=open('README.rst').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'
The documentation has lagged behind the code. New features should be documented.
Django 1.9 got rid of ValuesQuerySet which causes errors when djqscsv attempts to import it.
This is a public function and part of the API - it needs documentation and an example of how to use it.
Excel can't read unicode csvs unless there's a BOM in front of them. See http://stackoverflow.com/questions/5838605/python-dictwriter-writing-utf-8-encoded-csv-files
Resource interpreted as Document but transferred with MIME type text/csv: <path_to_file.csv>
I'd like to generate a CSV file not only with attributes, but also properties. The model could look like this:
class WorkingTime(models.Model):
"""\
Used for time tracking on projects.
"""
employee = models.ForeignKey(Employee)
start_time = models.DateTimeField(default=datetime.datetime.now())
end_time = models.DateTimeField(default=datetime.datetime.now(), blank=True, null=True)
def _delta(self):
"""\
Returns the difference between start and end time of the logged work.
"""
return self.end_time - self.start_time
delta = property(_delta)
Now I'd like to generate the CSV like this:
# make query
csv_qs = csv_qs.values(
'start_time',
'end_time',
'employee__last_name',
'employee__first_name',
'delta'
)
return render_to_csv_response(
csv_qs,
field_header_map={
'start_time': _(u'Begin'),
'end_time': _(u'End'),
'employee__last_name': _(u'Nachname'),
'employee__first_name': _(u'Vorname'),
'delta': _(u'Arbeitszeit'),
},
append_datestamp=True,
filename="Arbeitszeiten",
delimiter=';')
Is that somehow possible? Thanks!
Problem with:
í 237 u’\xed’ \xc3\xad \xed Latin small i with acute
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/core/handlers/base.py", line 114, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/views/generic/base.py", line 69, in view
return self.dispatch(request, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/views/generic/base.py", line 87, in dispatch
return handler(request, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/utils/decorators.py", line 29, in _wrapper
return bound_func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 22, in _wrapped_view
return view_func(request, *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/django/utils/decorators.py", line 25, in bound_func
return func(self, *args2, **kwargs2)
File "/Users/xpalacin/Sites/django_guiaterapeutica/main/views.py", line 148, in get
text = render_to_csv_response(queryset)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/djqscsv/djqscsv.py", line 49, in render_to_csv_response
write_csv(queryset, response, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/djqscsv/djqscsv.py", line 122, in write_csv
writer.writerow(merged_header_map)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/csv.py", line 148, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xed' in position 1: ordinal not in range(128)
The filename should be able to be called anything.
Related, the _append_datestamp
function should be updated to use os.splitext
instead of just truncating a fixed length from the end of the filename.
If there is a parameter in function render_to_csv_response(), which I can modify the filename of CSV generated ?
import djqscsv
data=...
djqcsv.render_to_csv_response(data)
Thanks!
Can you guys provide a example on Class based views
I am using djqscsv.render_to_csv_response()
function to export my django querset to *.csv
file. when I canceled downloading the *.csv
file, I got error like that:
Exception happened during processing of request from ('127.0.0.1', 61041)
[12/Oct/2018 17:54:05] "POST /accounts/data_extraction/ HTTP/1.1" 200 2375804
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 138, in run
self.finish_response()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 279, in write
self._write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 453, in _write
result = self.stdout.write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socket.py", line 594, in write
return self._sock.send(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
[12/Oct/2018 17:54:05] "POST /accounts/data_extraction/ HTTP/1.1" 500 59
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 138, in run
self.finish_response()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 279, in write
self._write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 453, in _write
result = self.stdout.write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socket.py", line 594, in write
return self._sock.send(b)
ConnectionAbortedError: [WinError 10053] An established connection was aborted by the software in your host machine
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 141, in run
self.handle_error()
File "D:\Company projects\websystem\truckcaliber\lib\site-packages\django\core\servers\basehttp.py", line 88, in handle_error
super(ServerHandler, self).handle_error()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 368, in handle_error
self.finish_response()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 180, in finish_response
self.write(data)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 274, in write
self.send_headers()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 625, in process_request_thread
self.finish_request(request, client_address)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 354, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\socketserver.py", line 681, in __init__
self.handle()
File "D:\Company projects\websystem\truckcaliber\lib\site-packages\django\core\servers\basehttp.py", line 155, in handle
handler.run(self.server.get_app())
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\handlers.py", line 144, in run
self.close()
File "c:\users\user\appdata\local\programs\python\python35-32\Lib\wsgiref\simple_server.py", line 36, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
What is wrong there?
This should be covered by annotating a queryset and then exporting it, but opening an issue to officially support this and add test coverage for it.
It looks like Following example at Usage section in README causes error on python3.5 environment.
If you need to write the CSV to a file you can use write_csv instead:
from djqscsv import write_csv
qs = Foo.objects.filter(bar=True).values('id', 'bar')
with open('foo.csv', 'w') as csv_file:
write_csv(qs, csv_file)
TypeError: write() argument must be str, not bytes
write_csv method trying to write byte (BOM) to file, so We need to open file as binary mode.
with open('foo.csv', 'wb') as csv_file:
write_csv(qs, csv_file)
I'm glad if you test those examples in your environment and confirm whether correction works.
Kind Regards,
Asayu123
OTM2 uses a copy+pasted version of _sanitize_unicode_record to export JSON values. Consider unit testing this function and publicly exporting it.
Hi, this library works great for me with one exception: when I save the download and look in Explorer to open the file, it just shows up as a generic file (not a csv). The only file type option to save the file is "all files". This is confirmed by examining the file's properties. If I manually add the file extension by editing the file name, then the file opens just fine.
This error occurs whether I specify a file name or not.
On the initial download pop up (where it asks you to either save or open), it shows file type as "application/vnd.ms-excel".
Any help with this would be greatly appreciated. Thanks.
This issue was raised to me from a user by email.
I found when writing utf-8 output both keys and values had to be encoded in the _sanitize_unicode_record function. Here's the implementation i used, which may be slightly redundant:
def _sanitize_unicode_record(record):
return {unicode(k): unicode(v).encode('utf-8') for k, v in record.items()}
These parameters are currently undocumented and have caused some confusion.
Hi,
Is it possible to render a csv with post data from a form? It looks like the view is using a GET response, but is there a way to combine the two?
Hi azavea:
First of all, thanks for the work, it really made my life easier! However, I still wonder if you guys have any plans to support m2m fields. I know it's tricky because django's Model.objects.values() doesn't play very well with m2m fields, and it's also tricky to handle which field in the m2m model you want to use for display. If you guys have some thoughts about it, like putting together the unicode for each m2m entry, etc, please let me know. Thanks!
2c86170 broke code coverage for some reason
I pass in the name of an event
object to the filename
keyword in render_to_csv_response()
. If the title
attribute of the event
contains a .
then I get the error that the file extension must be .csv
.
I can just do a event.title.replace('.', '')
before passing it in, but I am wondering why not include this line in the _validate_and_clean_filename()
function to sanitize the input?
I can submit a PR for it if you are ok with the idea
This is likely to be a better performing default for most people than a standard HttpResponse
, but since StreamingHttpResponse
is not a subclass of HttpResponse
and has a different API, making it the default would be a breaking change (and we would add a non_streaming
option to restore the old behaviour).
We could instead make it an optional feature by adding a render_to_streaming_csv_response
function or adding an optional streaming
option to render_csv_to_response
.
See the Django docs for a general overview of StreamingHttpResponse
and how to use it with CSVs
There seems to be no obvious way to give a format like %Y-%m-%d
or %Y-%m-%d - %H:%M
for a date or datetime field.
When pushing tags, Travis CI automatically pushes a new release of the library to PyPI.
This is working correctly, but we are trying to push artifacts for every combination in our build matrix, which causes all builds in the matrix after the first to fail, since the release is already on PyPI.
We should restrict our deploy
step to only run once, if possible. Travis CI build stages might be helpful here: https://docs.travis-ci.com/user/build-stages
This isn't actively harmful, since pushing artifacts to PyPI is the last step in the Travis CI test script, and we will separately test the master
branch at the same time, but it would be nice to fix this.
Is there a way to use the Model.get_FOO_display()
method to use the human-readable values for a choices field in the CSV?
This method is available in Django templates, as documented here: https://docs.djangoproject.com/en/1.11/ref/models/instances/#django.db.models.Model.get_FOO_display
If this doesn't already exist, if you point me in the right direction, I can try my hand at a PR.
This blog post is referenced on PIP and in the README, and is referenced as the source of documentation.
https://www.azavea.com/blog/2014/03/11/exporting-django-querysets-to-csv/
Can i choose delimiter?
My queryset has fields in a .extra
clause. I want the exported CSV to use a different column name for those fields. But if I specify that renaming via field_header_map
it doesn't work--the column header uses the original field name rather than the name I specified.
Where does it generate the CSV file with the model data?
Hey guys, thank you for the py3 support.
May I ask for the new pip package.
Thanks
Ok, now I'm just bikeshedding, but the package name is terrible
qs_csv
or something would be much more easier to stomach :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.