neurodata / ndstore Goto Github PK
View Code? Open in Web Editor NEWcode for storing neurodata images and image annotations
Home Page: http://neurodata.io
License: Apache License 2.0
code for storing neurodata images and image annotations
Home Page: http://neurodata.io
License: Apache License 2.0
I don't know enough about how the probabilitymap stuff was written to make any headway on this...
[16/May/2015 19:23:41] ERROR [ocp:views:86] Unknown exception in getCutout.
Traceback (most recent call last):
File "./ocpca/views.py", line 71, in cutout
django.http.HttpResponse(ocpcarest.putCutout(webargs,request.body))
File "/var/www/ocp/open-connectome/ocpca/ocpcarest.py", line 569, in putCutout
return selectPost ( rangeargs, proj, db, postdata )
File "/var/www/ocp/open-connectome/ocpca/ocpcarest.py", line 488, in selectPost
if voxarray.dtype == ocpcaproj.OCP_dtypetonp[ch.getDataType()]:
KeyError: u'float32'
Looks like OCP_dtypetonp wasn't updated for the new schema?
OCP_dtypetonp = {'uint8':np.uint8,'uint16':np.uint16,'uint32':np.uint32,'rgb32':np.uint32,'rgb64':np.uint64,'probability':np.float32
Or I'm just confused...
See ocpcarest line 1444. Looks like it's trying to pull getPropagate values from the project instead of the channel.
I have implemented the isotropic scaling hierarchy. This means changing the ocpconfig datasets and projects table. I need the current version pushed ASAP, so that I can integrate these changes. I don't need them pushed to the master branch. But, I do need them.
Format should be token/channel/plane/cutout
Fixed in d4e015e
@randalburns @kunallillaney: Did interface for probability overlays change? They quit working for me (actually about a month ago...)
Doesn't work (prob32)
http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/
Does work (anno32)
http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_synapseTruth/xy/1/5472,6496/8712,9736/1005/
----- 500 error, not particularly informative to me -----
HTTPError at /overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/
HTTP Error 500: INTERNAL SERVER ERROR
Request Method: GET
Request URL: http://openconnecto.me/ocp/overlay/0.7/kasthuri11_ac3_membranes/xy/1/5472,6496/8712,9736/1005/
Django Version: 1.6.1
Exception Type: HTTPError
Exception Value:
HTTP Error 500: INTERNAL SERVER ERROR
Exception Location: /usr/local/lib/python2.7/urllib2.py in http_error_default, line 527
Python Executable: /usr/share/ocp/virtualenvs/ocp/bin/uwsgi
Python Version: 2.7.3
Python Path:
['.',
'',
'/usr/share/ocp/virtualenvs/ocp/lib/python27.zip',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/plat-linux2',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-tk',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-old',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7',
'/usr/local/lib/python2.7/plat-linux2',
'/usr/local/lib/python2.7/lib-tk',
'/usr/share/ocp/virtualenvs/ocp/lib/python2.7/site-packages',
'/usr/share/ocp/open-connectome/django',
'/usr/share/ocp/open-connectome/util',
'/usr/share/ocp/open-connectome/ocpca',
'/usr/share/ocp/open-connectome/ocplib']
Server time: Tue, 28 Apr 2015 00:17:16 -0400
it would be great to have a service such that when we upload a brain mask,
server side that mask is propagated both up & down zoom levels,
and applied, which automatically builds a new image token that is the same as the original one but appended with '_mask', or some such
for each project, i was thinking that perhaps we can have a field called 'parent',
which either points to another ocp token, or is root.
in particular, if the project is an annotation project, then its parent is the image data.
similarly, for a color corrected project.
for the raw images, its field is root.
this makes keeping track of things way easier, and doesn't require a full LIMS.
@WillGray @randalburns @kunallillaney @alexbaden - thoughts?
i am thinking it would be great to automatically generate a table from projinfo (for public projects) that had fields like:
@kunallillaney @randalburns @alexbaden @WillGray how hard is that to make?
for example, 'default-channel' is not ok!
The file appears to post fine, but when I download the cutout in HDF5 format using the same URL, I get a file of all "0s".
[04/Jun/2015 16:23:25] ERROR [ocp:views:84] Unknown exception in getCutout.
Traceback (most recent call last):
File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 69, in cutout
django.http.HttpResponse(ocpcarest.putCutout(webargs,request.body))
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 559, in putCutout
return selectPost ( rangeargs, proj, db, postdata )
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 530, in selectPost
return str(entityid)
UnboundLocalError: local variable 'entityid' referenced before assignment
[04/Jun/2015 16:23:25] ERROR [django.request:base:231] Internal Server Error: /ocp/ca/apiUnitTests/apiUnitTestKasthuriProb/hdf5/1/2560,3584/1000,2024/64,112/overwrite/
Traceback (most recent call last):
File "/Users/abaden/.virtualenvs/ocp/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 85, in cutout
raise OCPCAError("Unknow exception in getCutout")
OCPCAError: 'Unknow exception in getCutout'
Based on Catmaid? @jovo any ideas?
Details based on our discussion are here:
http://w.ocp.me/private:graph_construction
Open to other ideas - basic functionality (v1) seems sufficient for now, although propagate would also be nice.
Right now it produces DATATYPE, which contains one of {image, annotation, probmap, etc}. I'd like to rename that DBTYPE and add DATATYPE, which contains {uint8, uint16, uint32, etc}. We could probably engineer a solution to pull the DATATYPE from the CUTOUT, but this is a more general way to do it.
yields the following message
[23/Jun/2015 20:40:26] WARNING [ocp:ocpcarest:203] Incorrect arguments for imgSlice apiUnitTestKasthuri/xy/1/1200,1400/2300,2500/557/filter/3653/. 'NoneType' object has no attribute 'groups'
OCP problem or new syntax?
http://w.ocp.me/images has a bunch of fields that i'd like in ocp.
i'm sure @kunallillaney working with the HBP people knows of others that they need/want.
when creating a new database, syncdb needs to be run twice to create the database. why?
Backend + API - this may be equivalent to 8-bit image upload, or nearly so...
Many times we don't need 32-bit precision on probabilities; as datasets scale, we could significantly cut down on data being transmitted/written. Two near term use cases (cell detection and large scale synapses)
Two errors:
1192 offset = proj.datasetcfg.offset[resolution]
1193 corner = (h5f['XYZOFFSET'][0]-offset[0],h5f['XYZOFFSET'][1]-offset[1],h5f[' XYZOFFSET'][2]-offset[2])
1194 dim = h5f['CUTOUTSIZE'][:]
1195 resolution = h5f['RESOLUTION'][0]
Moving the resolution (line 1195) up to line 1191 seemed to fix that problem (did not commit). But then I got this error:
Traceback (most recent call last):
File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 139, in queryObjects
return django.http.HttpResponse(ocpcarest.queryAnnoObjects(webargs,request.body), content_type="product/hdf5")
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 1197, in queryAnnoObjects
if not proj.datasetcfg.checkCube( resolution, corner[0], corner[0]+dim[0], corner[1], corner[1]+dim[1], corner[2], corner[2]+dim[2] ):
TypeError: checkCube() takes at most 5 arguments (8 given)
I think Kunal already fixed this in a previous query?
Randal, We need a page/interface via which a user can create channels while creating a project.
It seems like it should still delete the project. Perhaps the user already deleted the database?
The default channel bug is not yet resolved. When creating a channel, if we select the default channel button, can lead to multiple default channels.
Similarly when deleting a default channel, it does not convert an existing channel to a default channel leading to a state where there might be channels but no default channel.
Center of map XY Coordinates (& ability to change them)
Ability to specify and navigate to a zindex
@alexbaden @randalburns @kunallillaney
they want to be able to build a plugin to pull data from OCP.
unclear which spec they want to pull down, they want us to export a vaa3d file.
they will send us the details.
I have created a local django server (open-connectome) and the server passes all tests (from py.test). However when attempting to log into the server (/accounts/login/) I am getting a "CSRF verification failed" error.
Allows a user to link to existing channel tables instead of creating channel tables.
Fix the the activation email module. The current setup is for django1.6 and broken because it is hard-coded to send all emails with link for dsp061.pha.jhu.edu. I think django1.7 does it differently(https://docs.djangoproject.com/en/dev/topics/email/)
Delete channel does not remove the tables. I have added a new method in OCPCADB called deleteOCPCAChannel() which needs to be called for channel deletes. I am calling this from views.py but seems to throw some error. ( the function in itself is tested and works fine)
Login Protect /channel and /token pages.
Non-owners of data public dataset should not be able to update it.
Non public datasets are not available to super-users.
Lock the datatype for a channel type, image can only be uint8/uint16, annotation can be uint32/uint64 and so on.
Lock the propagation state to either propagated or non-propagated. It is set to propagating at start then it will never leave that state via the service.
If a user creates a project and set a default token, then the public value of token should be the same as that of the project, now all default tokens are public.
If a project is a public should all it's tokens be public? If no then we need to restrict this, issue 1 and this are related and have the same fix.
Ensure that there exits exactly one default channel.
Create a "link to existing channel" option in the GUI which does not create the channel tables.
It doesn't seem like it would be too hard... just add a check to see if the h5py.get call returns None, and then throw an error. I'm happy to do it. @randalburns @kunallillaney, what say you?
Two test fail:
Shave test - Cannot post empty cubes to the database(all zeros)
Second test - Changing the HDF5 file format for posting data.
x and y tiles were swapped in the URL arguments. Works now!
See fed4e6c for the fix.
Look at a new window function. The file is ocpca/windowcutout.py
document how to use thunder+ocp
When calling /ocp/ca/kharris15apical/annos/id/3/582/530/88/ I get the following:
[09/Jun/2015 18:46:24] "GET //ocp/ca/kharris15apical/annos/id/3/582/530/88/ HTTP/1.1" 404 3431
[09/Jun/2015 18:46:28] ERROR [ocp:views:86] Unknown exception in getCutout.
Traceback (most recent call last):
File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 62, in cutout
return django.http.HttpResponse(ocpcarest.getCutout(webargs))
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 547, in getCutout
return selectService ( service, webargs, proj, db )
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 388, in selectService
return annId ( webargs, proj, db )
File "/Users/abaden/Projects/open-connectome/ocpca/ocpcarest.py", line 346, in annId
(resolution, voxel) = restargs.voxel ( imageargs, proj.datasetcfg )
File "/Users/abaden/Projects/open-connectome/util/restargs.py", line 147, in voxel
if not ( datasetcfg.checkCube ( resolution, [x,y,z], [x+1,y+1,z+1] )):
TypeError: checkCube() takes exactly 5 arguments (4 given)
[09/Jun/2015 18:46:28] ERROR [django.request:base:231] Internal Server Error: /ocp/ca/kharris15apical/annos/id/3/582/530/88/
Traceback (most recent call last):
File "/Users/abaden/.virtualenvs/ocp/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/abaden/Projects/open-connectome/django/ocpca/views.py", line 87, in cutout
raise OCPCAError("Unknow exception in getCutout")
OCPCAError: 'Unknow exception in getCutout'
[09/Jun/2015 18:46:28] "GET /ocp/ca/kharris15apical/annos/id/3/582/530/88/ HTTP/1.1" 500 62955
Not sure what exactly is going on. Feel free to test on bv1.
@randalburns @kunallillaney @WillGray @gkiar
http://research.mssm.edu/cnic/swc.html
is the format that vaa3d and janelia are now using to store skeletons
create file names for HDF5 downloads
What's our strategy for users to get tokens/projects?
Do we want to document the web interface and allow anyone to make an account and request? Or do you prefer we point them to a person or a support page somewhere for now?
Definitely getting to the point that lots of people need to make these. I'd love it if we could do this in the API - but would need some sort of user auth.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.