Bulk collection occurrence doesn't have coordinates. These need to be added.

for the following

• PBDB
• ABCD EFG

Production appears to still be in debug mode as of 5/5/2014

The point_x method returns the latitude when it should return longitude and vice-verse for point_y

A consequence of moving the data from Carmen to TACC is that we simplify the database system, everything is now in one database and we don't need separate database routers in django. The downside is that we break the GIS connections setup for carmen. We need to allow direct access to the database at least for the GIS workstation in SAC 5.186. WFS is not very reliable. It currently is not working on QGIS from SAC 5.186. Also to make WFS connections in ArcGIS we need the ESRI Data Interoperability extension (ugh!).

We remodeled the drp app in this way and we need to do the same for mlp. Create a Biology class that is a subclass of Occurrence. Eventually we need to do the same for Archaeology and Geology as well.

The custom save method in drp.models.drp_biology raises an IndexError when the biology table is empty, as it is when the test database is created for unit testing.

Provide a summary of occurrence objects on summary page for each project. This could include 1) a tally of all occurrences with a breakdown of observations and collected fossils 2) an abundance chart of all taxa MNI and NISP 3) A map of fossil densities in the study area.

Add rank label before taxonomic identification, and order elements in drop down list.

it is returning some sort of dictionary error, because occurrences with no entry in biology have far fewer fields than occurrences with biology entries, and errors occur in the csv writer

For MLP and DRP we have updated taxonomy lookups so there is now a consistent taxonomic tree for both projects. To accomplish this taxonomy field in the biology table is a foreign key to the taxonomy tables. We need to update the display of this field in the item detail view to provide a nicer presentation and to simplify the drop down list.

Shannon McPherron created a sites project app that maps locations of MP archaeology sites. Need to integrate this app into the project, but develop first on a new branch.

Their appear to be several orphaned drp_biology records with no corresponding drp_occurrence records. By default Django should be performing cascade delete on drp_biology when an associated occurrence record is deleted. Check that this is the case. Test for orphaned biology records.

@wabarr will be in charge of creating secrets.py file to deal with this

Attempting to delete and occurrence in the admin produces an Attribute error,
'NoneType' object has no attribute 'replace'

Bug arises because of broken unicode method in mlp.Occurrence data model

Create a suite of tests for the index view of the data ap

With permissions set for everyone

The drp data initially resided on the carmen.la.utexas.edu server. Those data have moved to TACC servers and there's no need for the DB router section in the code base. An advantage to keeping this code is that it demonstrate how to implement the DB routing even though we no longer need it. Leaving the code active forces the test suite to create separate test databases and this slows down testing and complicates testing. For this reason we should comment out the routing code, but leave it in the code base as a guide in case we need to access external databases in the future.

port existing functionality

At meeting on Jan 31, @dennereed said he was going to set up the base app with

• fiber
• postGIS
• hitting carmen
• templates

Currently, using the standard DjangoAuthorization() class, any authenticated user can GET data from the drp_occurrence api. This is not the desired behaviour....I need to override the DjangoAuthorization() class to fix this

http://www.paleocore.org/home/ Does not redirect to http://www.paleocore.org as it should. Instead, it just displays a blank page.

Personally, I recommend taking out /home/ from the urls entirely. Is there a purpose for the /home/ link?

read from schema app database

by default, there is nothing to prevent brute force login attacks. There is a very simple package called django-axes that takes care of this. Need to implement

Need a dedicated app to resolve taxonomic relationships. Use Django Ajax Selects. This would be most useful for populating item_scientific_name.

Enter coords from notes or GPS locations.

Downloading to csv action in occurrence admin raises attribute error. See messages below.

Perhaps caused by occurrence instances that lack a geom? This should not happen, but need exception handling for the event that they do. Need to locate which points lack geom and add exception handling in admin.py method at line 170.

AttributeError at /admin/mlp/occurrence/
'NoneType' object has no attribute 'get_x'
Request Method: POST
Request URL: http://localhost:8000/admin/mlp/occurrence/
Django Version: 1.6.5
Exception Type: AttributeError
Exception Value:
'NoneType' object has no attribute 'get_x'
Exception Location: /Users/reedd/Dropbox/pycharm/paleocore/mlp/admin.py in create_data_csv, line 170
Python Executable: /usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Python Version: 2.7.6
Python Path:
['/Users/reedd/Dropbox/pycharm/paleocore',
'/Library/Frameworks/SQLite3.framework/Versions/B/Python/2.7',
'/Library/Frameworks/GEOS.framework/Versions/3/Python/2.7',
'/Library/Python/2.7/site-packages/numpy-override',
'/Library/Python/2.7/site-packages/matplotlib-override',
'/Library/Frameworks/GDAL.framework/Versions/1.11/Python/2.7/site-packages',
'/Users/reedd/Dropbox/pycharm/paleocore',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
'/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages',
'/Library/Python/2.7/site-packages']
Server time: Mon, 8 Dec 2014 15:48:58 -0600

API to R, parsing URL parameters and returning CSV content

Each project is its own app and in addition there is a data app. The url to each project is via the data app, e.g. paleocore/data/mlp. We added some functionality directly under the project app, e.g. paleocore/mlp/upload. So there is a confusing set of urls. This needs to be sorted out. We need to decide what path is most appropriate and be consistent.

We need a data export/import workflow for each project. A safe and controlled way to export data for field data collection and a similar mechanism to import occurrence data back in.

• Finalize the API for the turkana data set
• Document the code with comments
• Write tests for the API
• Write a fiber page documenting how to pull data with the API in R, QGIS, SQL, Web.

only for projects which currently have data

• DRP
• MLP
• Turkana
• Omo

Currently, mlp.views class UploadKMLView.form_valid includes attempts to validate field numbers using datetime.strptime(attributes_dict.get("Time"), "%b %d, %Y, %H:%M %p"). For some reason the %p is not read and PM times are recorded as AM.

This item needs a barcode number added to the DB. Barcode number should be 1404. See email thread below....

If we are sure that the photo I sent matches your photo of MLP-1404, then we can resolve the problem. It appears that the point did not get a barcode entered into the database in the field. So the point exists (with coordinates) but does not have a barcode.

Denne: this point is ID# 1189 in the database. It currently has a barcode of 0 but should be changed to 1404. That will solve the problem.

On Wed Dec 10 2014 at 10:19:02 AM Denis Geraads [email protected] wrote:
yes, it is this one (photo attached). I don't know why there are no coordinates, but since this is apparently the only real problem in the database, we can leave it for now.
I attach the January and November updated catalogues (not fused because coord are in LatLon vs. UTM)
Best
D
Denis,

It looks like it didn't get entered in, as there is no barcode matching 1404. I looked for fossils that lack a barcode, and I found this one. Is this the tooth (photo attached)?

On Wed Dec 10 2014 at 7:11:41 AM Denis Geraads [email protected] wrote:
Hi Andrew
Thanks for the photos.
I am updating the catalogue with my observations in the Museum. No major problem, except that I do not have coordinates for MLP-1404, a suid premolar. I am sure of the number, because I have a photo of the lower part of the tag with the bar code, but not of the upper part with the coordinates...
Thanks

Denis

Create unit tests for drp app

http://paleocore.org/squiggly_piggly/ returns a fiber page with the old default template rather than 404. This happens only in production, not in dev.

Using the search window in the admin view for drp_biology raises an error in production.

FieldError at /admin/drp/drp_biology/
Cannot resolve keyword 'lowest_level_identification' into field.

From the Django 1.2 docs
dict on Model instances
Historically, the dict attribute of a model instance has only contained attributes corresponding to the fields on a model.

In order to support multiple database configurations, Django 1.2 has added a _state attribute to object instances. This attribute will appear in dict for a model instance. If your code relies on iterating over dict to obtain a list of fields, you must now filter the _state attribute of out dict.

The download to .csv admin action on the mlp and drp databases includes _state field that returns a ModelState object. This should not be included in the download. Update the admin to remove this.

Unit tests for mlp app

Should we enable detailed logging (which django has the capability to do) and/or create an "actions" table that records the following whenever a database transaction occurs?

User       DateTime            Action     Record
wabarr    05/06/14 15:46       delete     drp_occurrence.id = 999

We urgently need to reconfigure the db setup to allow unit testing in paleocore. The issue seems to be that the test suite cannot generate a testing drp db on carmen because this db lives in the sde schema. Django does not recognize schemas and cannot create a test db inside that schema.

I think the solution is to move the db to the public schema on carmen thereby retaining the multi-db structure (for now). That should allow django to create the test DB.

If that doesn't work then we need to more the drp db to the TACC servers

Scientific names for biology instances are stored in multiple places across at least two tables; the occurrence table has scientificName and then there are several fields in the biology table dedicated to storing names for different parts of the taxonomic hierarchy. With this comes the risk that the two (or more) areas don't match. We need some sort of automated approach to tracking taxonomic assignments for occurrence instances.

Pycharm shows an error that reads
"Signature of method Biology.save() does not match signature of base method in class 'Model'" The same error is showing in drp. Taxonomic updates may be broken there as well.