chrisspen / django-sec Goto Github PK

any way you can put up a better explanation of how to use this package?

1. Parsing all the data for a specific year seems to take forever (more than a week). Is this normal ? Is there any way the script could run faster by downloading all the files at the beginning and parse the data in several files at the same time ?
2. Exporting the data to excel seem broken.

I had to alter: https://github.com/chrisspen/django-sec/blob/master/django_sec/migrations/0001_initial.py#L63

By removing the seemingly erroneous 'b' preceding the 'ticker' string.

When using the Company search, Django throws an exception, as _ticker is activated for search but isn't contained in the company object. I fixed the issue in the file i was working on by commenting out line 211 in the admin.py file.

Hello Chriss,
I am getting an invalid syntax now when I install from pip, it seems to be just some extra tabs, it doesn't affect the build but best to remove them .

Found existing installation: django-sec 0.3.3
Uninstalling django-sec:
Successfully uninstalled django-sec
Running setup.py install for django-sec
File "/usr/local/lib/python2.7/dist-packages/django_sec/management/commands/sec_mark_units.py", line 75

    ^
SyntaxError: invalid syntax


warning: no files found matching '*' under directory 'chroniker/tests/fixtures'

Successfully installed django-sec
Cleaning up...

Hello,
I think this is another foreign key issue introduced in Django 1.9.
If you go on : /admin/django_sec/company/
Put a company name say Verdier I get:

FieldError at /admin/django_sec/company/
Cannot resolve keyword '_ticker' into field. Choices are: attributes, cik, filings, load, max_date, min_date, name
Request Method: GET
Request URL: http://localhost:8000/admin/django_sec/company/?q=Verdier
Django Version: 1.9
Exception Type: FieldError
Exception Value:
Cannot resolve keyword '_ticker' into field. Choices are: attributes, cik, filings, load, max_date, min_date, name
Exception Location: /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py in names_to_path, line 1330
Python Executable: /usr/bin/python
Python Version: 2.7.9
Python Path:
['/home/pi/Documents/DjangoSec',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-arm-linux-gnueabihf',
'/usr/lib/python2.7/lib-tk',
'/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7']
Server time: Mon, 2 May 2016 22:05:30 +0000

Is the use of WGET nessesary (maybe for the parallel processing)? I have rewritten to use 'requests' module instead. I am on windows and don´t want to install WGET.

SEC has closed the FTP download option with end of 2016. The index download uses this option. This means that 'sec_import_index.py' line 129 should be:

url = 'https://www.sec.gov/Archives/edgar/full-index/%d/QTR%d/company.zip' % (year, quarter)

I have been trying to extract attributes from 10-K filings but I dont see many balance sheet fields (like assets) extracted even though I can see those fields in /django_sec/fixtures/normalized_fields.csv. How can I extract the balance sheet fields?

I noticed that sec_import_index hang when downloading years 2016,2015,2014,2013 qtr2 etc. Strange thing is it will work for qrt 1 of 2013...

`C:\Users\Math\math_sec>python manage.py sec_import_index --start-year=2013 --end-year=2016
Opening index file /tmp/django_sec\company_2013_1.zip.
Found 475 prior ciks.
Processing record 303583 of 303583 (100.00%).

73537 new companies found.
301393 new indexes found.
Downloading ftp://ftp.sec.gov/edgar/full-index/2013/QTR2/company.zip.
Opening index file /tmp/django_sec\company_2013_2.zip.
Traceback (most recent call last):
File "manage.py", line 22, in
execute_from_command_line(sys.argv)
File "C:\Python27\lib\site-packages\django\core\management_init_.py", line 367, in execute_from_command_line
utility.execute()
File "C:\Python27\lib\site-packages\django\core\management_init_.py", line 359, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 294, in run_from_argv
self.execute(*args, **cmd_options)
File "C:\Python27\lib\site-packages\django\core\management\base.py", line 345, in execute
output = self.handle(*args, **options)
File "C:\Python27\lib\site-packages\django_sec\management\commands\sec_import_index.py", line 111, in handle
max_lines=max_lines)
File "C:\Python27\lib\site-packages\django_sec\management\commands\sec_import_index.py", line 163, in get_filing_list
zipf = ZipFile(fn)
File "C:\Python27\lib\zipfile.py", line 770, in init
self._RealGetContents()
File "C:\Python27\lib\zipfile.py", line 811, in _RealGetContents
raise BadZipfile, "File is not a zip file"
zipfile.BadZipfile: File is not a zip file`

Hello,

I can run the server and see the webpage, but I can not use the sec_import_index.

Here is the error I get :
C:\Users\Math\math_sec>python manage.py sec_import_index --start-year=2015 --end-year=2016 Traceback (most recent call last): File "manage.py", line 22, in <module> execute_from_command_line(sys.argv) File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 367, in execute_from_command_line utility.execute() File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 359, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 208, in fetch_command klass = load_command_class(app_name, subcommand) File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 40, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "C:\Python27\lib\importlib\__init__.py", line 37, in import_module __import__(name) File "C:\Python27\lib\site-packages\django_sec\management\commands\sec_import_index.py", line 12, in <module> from django.core.management.base import NoArgsCommand ImportError: cannot import name NoArgsCommand

Any way to fix this

Hello there,
I think there is another bug potentially, I only have marked one company for download but when I run:

python manage.py sec_import_attrs --start-year=2016 --end-year=2017 --form=10-Q,10-K

Running process: {'start_year': '2016', 'show_pending': False, 'force': False, 'verbose': False, 'settings': None, 'cik': None, 'pythonpath': None, 'verbosity': 1, 'traceback': None, 'multi': 0, 'forms': '10-Q,10-K', 'no_color': False, 'end_year': '2017', 'quarter': None}
total_count: 7114
None [ ] 0 of ? ?% eta=?: 7114 total rows.
None [ ] 1 of 7114 0% eta=2016-05-05 05:14:06.056550+00:00: Processing index edgar/data/1006045/0001564590-16-015763.txt.

It's parsing every company in the database.